[Xenomai-core] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Xenomai-core] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Peter Soetens]

Just in case you hooked off the long discussion about the issues we found from
Xenomai 2.3.2 on:

  o We are using the xeno_native skin, create Xeno tasks and semaphores, but 
have strong indications that the crashes are caused by the memory allocation 
scheme of Xenomai in combination with task creation/deletion
  o We found two ways to break Xenomai, causing a 'Killed' (rt_task_delete) 
and causing an OOPS (rt_task_join).
  o They happen on 2.6.20 and 2.6.22 kernels
  o On the 2.3 branch, r2429 works, r2433 causes the faults. The patch is 
small, and in the ChangLog: 

2007-05-11  Philippe Gerum  <rpm@xenomai.org>

    * include/nucleus/heap.h (xnfreesafe): Use xnpod_current_p() when
    checking for deferral.

    * include/nucleus/pod.h (xnpod_current_p): Give exec mode
    awareness to this predicate, checking for primary/secondary mode
    of shadows.

2007-05-11  Gilles Chanteperdrix  <gilles.chanteperdrix@xenomai.org>

    * ksrc/skins: Always defer thread memory release in deletion hook
    by calling xnheap_schedule_free() instead of xnfreesafe().

  o We reverted this patch on HEAD of the 2.3 branch, but got -ENOMEM errors 
during Xenomai resource allocations, indicating that later changes depend on 
this patch. So we use clean HEAD again further on to find the causes:
 o A first test (in Orocos) creates one thread, two semaphores, lets it wait 
on them and cleans up the thread.
 o During rt_task_delete, our program gets 'Killed' (without joinable thread), 
hence a user space problem. However, gdb is of no use, all thread info is 
lost.
 o We made the thread joinable (T_JOINABLE), and then joined. This bypassed 
the Kill on the first run but causes an OOPS the second time the same 
application is started:

Oops: 0000 [#1]
PREEMPT
CPU:    0
EIP:    0060:[<fef4a1f3>]    Not tainted VLI
EFLAGS: 00010002   (2.6.20.9-ipipe-1.8-08 #2)
EIP is at get_free_range+0x56/0x160 [xeno_nucleus]
eax: f3a81d01   ebx: 00000200   ecx: 55550101   edx: fef62b00
esi: 55550101   edi: 00000200   ebp: f0f33ec4   esp: f0f33e98
ds: 007b   es: 007b   ss: 0068
Process NonPeriodicActi (pid: 3020, ti=f0f32000 task=f7ce61b0 
task.ti=f0f32000)
Stack: 00000000 00000600 fef62b80 f3a81b24 f3a80000 fef62ba4 f3a80720 55550101
       00000600 f0f33f18 f7ce6360 f0f33ee4 fef4a948 fef62b80 f0f33f08 00000000
       00000400 f0f33f18 f7ce6360 f0f33f50 ff13e1de 00000282 00000282 bfab6350
Call Trace:
 [<c0103ffb>] show_trace_log_lvl+0x1f/0x35
 [<c01040bb>] show_stack_log_lvl+0xaa/0xcf
 [<c01042a9>] show_registers+0x1c9/0x392
 [<c0104588>] die+0x116/0x245
 [<c0110fca>] do_page_fault+0x287/0x61d
 [<c010ea35>] __ipipe_handle_exception+0x63/0x136
 [<c029466d>] error_code+0x79/0x88
 [<fef4a948>] xnheap_alloc+0x15b/0x17d [xeno_nucleus]
 [<ff13e1de>] __rt_task_create+0xe0/0x171 [xeno_native]
 [<fef5655f>] losyscall_event+0xaf/0x170 [xeno_nucleus]
 [<c0138804>] __ipipe_dispatch_event+0xc0/0x1da
 [<c010e90b>] __ipipe_syscall_root+0x43/0x10a
 [<c0102e79>] system_call+0x29/0x41
 =======================
Code: 74 61 85 c0 74 5d c7 45 e0 00 00 00 00 8b 4d e4 8b 49 10 89 4d ec 85 c9 
74 38 8b 45 dc 8b 78 0c 89 4d f0 89 ce 89 fb eb 02 89 ce <8b> 09 8d 04 3e 39 
c1 0f 94 c2 3b 5d d8 0f 92 c0 01 fb 84 c2 75
EIP: [<fef4a1f3>] get_free_range+0x56/0x160 [xeno_nucleus] SS:ESP 
0068:f0f33e98
[hard lockup]

  o Our application is also mixing the original RT_TASK struct and return 
value of the rt_task_self() function call when calling rt_ functions. 
Switching between one of those influences the crashing behaviour as well, not 
further investigated.

  o This was reproduced on two different systems (one with SMI workaround 
working)
 
You have the patch that broke things, I hope this gives you a hint on what 
causes our crashes. Know that Orocos as-is has worked with Xenomai from  
Xenomai 2.0 on.

Peter

-- 
Peter Soetens -- FMTC -- <http://www.fmtc.be>

Re: [Xenomai-core] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Philippe Gerum]

On Fri, 2007-09-07 at 11:27 +0200, Peter Soetens wrote:
> Just in case you hooked off the long discussion about the issues we found from
> Xenomai 2.3.2 on:
> 
>   o We are using the xeno_native skin, create Xeno tasks and semaphores, but 
> have strong indications that the crashes are caused by the memory allocation 
> scheme of Xenomai in combination with task creation/deletion
>   o We found two ways to break Xenomai, causing a 'Killed' (rt_task_delete) 
> and causing an OOPS (rt_task_join).
>   o They happen on 2.6.20 and 2.6.22 kernels
>   o On the 2.3 branch, r2429 works, r2433 causes the faults. The patch is 
> small, and in the ChangLog: 
> 
> 2007-05-11  Philippe Gerum  <rpm@xenomai.org>
> 
>     * include/nucleus/heap.h (xnfreesafe): Use xnpod_current_p() when
>     checking for deferral.
> 
>     * include/nucleus/pod.h (xnpod_current_p): Give exec mode
>     awareness to this predicate, checking for primary/secondary mode
>     of shadows.
> 
> 2007-05-11  Gilles Chanteperdrix  <gilles.chanteperdrix@xenomai.org>
> 
>     * ksrc/skins: Always defer thread memory release in deletion hook
>     by calling xnheap_schedule_free() instead of xnfreesafe().
> 
>   o We reverted this patch on HEAD of the 2.3 branch, but got -ENOMEM errors 
> during Xenomai resource allocations, indicating that later changes depend on 
> this patch. So we use clean HEAD again further on to find the causes:
>  o A first test (in Orocos) creates one thread, two semaphores, lets it wait 
> on them and cleans up the thread.

Please point me at the actual Orocos test code that breaks, with the
hope to get a fairly standalone test case from it; if you do have a
standalone test case already, this would be even better. I intend to
address this issue asap.

>  o During rt_task_delete, our program gets 'Killed' (without joinable thread), 
> hence a user space problem. However, gdb is of no use, all thread info is 
> lost.

SIGKILL is sent from the nucleus upon a call to rt_task_delete() which
targets a non-current task, in order to make sure this user-space task
will go away from a Linux context, since we don't want the kernel TCB
Xenomai maintains for it, to be wiped out before the mated userland
thread has really exited. IOW, this case boils down to an asynchronous
cancellation, where Linux is asked to kick out the target task first,
Xenomai then catches the event and cleans up the TCB on its side
afterwise.

As per POSIX, a lethal signal sent to a single thread zaps all other
threads belonging to the same process, which explains why your process
dies. We could be a bit smarter by handling this situation using a
hidden exit from a trapped signal handler, I guess.

>  o We made the thread joinable (T_JOINABLE), and then joined. This bypassed 
> the Kill on the first run but causes an OOPS the second time the same 
> application is started:
> 
> Oops: 0000 [#1]
> PREEMPT
> CPU:    0
> EIP:    0060:[<fef4a1f3>]    Not tainted VLI
> EFLAGS: 00010002   (2.6.20.9-ipipe-1.8-08 #2)
> EIP is at get_free_range+0x56/0x160 [xeno_nucleus]
> eax: f3a81d01   ebx: 00000200   ecx: 55550101   edx: fef62b00
> esi: 55550101   edi: 00000200   ebp: f0f33ec4   esp: f0f33e98
> ds: 007b   es: 007b   ss: 0068
> Process NonPeriodicActi (pid: 3020, ti=f0f32000 task=f7ce61b0 
> task.ti=f0f32000)
> Stack: 00000000 00000600 fef62b80 f3a81b24 f3a80000 fef62ba4 f3a80720 55550101
>        00000600 f0f33f18 f7ce6360 f0f33ee4 fef4a948 fef62b80 f0f33f08 00000000
>        00000400 f0f33f18 f7ce6360 f0f33f50 ff13e1de 00000282 00000282 bfab6350
> Call Trace:
>  [<c0103ffb>] show_trace_log_lvl+0x1f/0x35
>  [<c01040bb>] show_stack_log_lvl+0xaa/0xcf
>  [<c01042a9>] show_registers+0x1c9/0x392
>  [<c0104588>] die+0x116/0x245
>  [<c0110fca>] do_page_fault+0x287/0x61d
>  [<c010ea35>] __ipipe_handle_exception+0x63/0x136
>  [<c029466d>] error_code+0x79/0x88
>  [<fef4a948>] xnheap_alloc+0x15b/0x17d [xeno_nucleus]

The only explanation looking at this backtrace is that the system heap
has been corrupted by the previous exit; likely a side effect of the
deferral.

>  [<ff13e1de>] __rt_task_create+0xe0/0x171 [xeno_native]
>  [<fef5655f>] losyscall_event+0xaf/0x170 [xeno_nucleus]
>  [<c0138804>] __ipipe_dispatch_event+0xc0/0x1da
>  [<c010e90b>] __ipipe_syscall_root+0x43/0x10a
>  [<c0102e79>] system_call+0x29/0x41
>  =======================
> Code: 74 61 85 c0 74 5d c7 45 e0 00 00 00 00 8b 4d e4 8b 49 10 89 4d ec 85 c9 
> 74 38 8b 45 dc 8b 78 0c 89 4d f0 89 ce 89 fb eb 02 89 ce <8b> 09 8d 04 3e 39 
> c1 0f 94 c2 3b 5d d8 0f 92 c0 01 fb 84 c2 75
> EIP: [<fef4a1f3>] get_free_range+0x56/0x160 [xeno_nucleus] SS:ESP 
> 0068:f0f33e98
> [hard lockup]
> 
>   o Our application is also mixing the original RT_TASK struct and return 
> value of the rt_task_self() function call when calling rt_ functions. 
> Switching between one of those influences the crashing behaviour as well, not 
> further investigated.
> 

This should not make any difference regarding the bug above. Both
methods boil down to returning an abstract handler to the task, which
serves as an index to the in-kernel TCB maintained by Xenomai. In any
case, this handle is fully validated before any use, and does not even
carry any memory pointer semantics. We are playing with PTDs to maintain
the rt_task_self() information from userland instead of retrieving it
systematically from a syscall, but would the associated memory be
corrupted, the crash would then be confined to the userland process.

>   o This was reproduced on two different systems (one with SMI workaround 
> working)
>  
> You have the patch that broke things, I hope this gives you a hint on what 
> causes our crashes. Know that Orocos as-is has worked with Xenomai from  
> Xenomai 2.0 on.

> 
> Peter
> 
-- 
Philippe.

Re: [Xenomai-core] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Gilles Chanteperdrix]

Philippe Gerum wrote:
 > On Fri, 2007-09-07 at 11:27 +0200, Peter Soetens wrote:
 > > Just in case you hooked off the long discussion about the issues we found from
 > > Xenomai 2.3.2 on:
 > > 
 > >   o We are using the xeno_native skin, create Xeno tasks and semaphores, but 
 > > have strong indications that the crashes are caused by the memory allocation 
 > > scheme of Xenomai in combination with task creation/deletion
 > >   o We found two ways to break Xenomai, causing a 'Killed' (rt_task_delete) 
 > > and causing an OOPS (rt_task_join).
 > >   o They happen on 2.6.20 and 2.6.22 kernels
 > >   o On the 2.3 branch, r2429 works, r2433 causes the faults. The patch is 
 > > small, and in the ChangLog: 
 > > 
 > > 2007-05-11  Philippe Gerum  <rpm@xenomai.org>
 > > 
 > >     * include/nucleus/heap.h (xnfreesafe): Use xnpod_current_p() when
 > >     checking for deferral.
 > > 
 > >     * include/nucleus/pod.h (xnpod_current_p): Give exec mode
 > >     awareness to this predicate, checking for primary/secondary mode
 > >     of shadows.
 > > 
 > > 2007-05-11  Gilles Chanteperdrix  <gilles.chanteperdrix@xenomai.org>
 > > 
 > >     * ksrc/skins: Always defer thread memory release in deletion hook
 > >     by calling xnheap_schedule_free() instead of xnfreesafe().
 > > 
 > >   o We reverted this patch on HEAD of the 2.3 branch, but got -ENOMEM errors 
 > > during Xenomai resource allocations, indicating that later changes depend on 
 > > this patch. So we use clean HEAD again further on to find the causes:
 > >  o A first test (in Orocos) creates one thread, two semaphores, lets it wait 
 > > on them and cleans up the thread.
 > 
 > Please point me at the actual Orocos test code that breaks, with the
 > hope to get a fairly standalone test case from it; if you do have a
 > standalone test case already, this would be even better. I intend to
 > address this issue asap.

Before you have a piece of code that causes the crash, I gave a look at
the code involved. The only suspicious thing I see is that the correct
working of native skins thread termination depends on the execution
order of the two deletion hooks, the one in task.c and the one in
syscall.c. As a matter of fact, if the one in task.c is executed before
the one in syscall.c, the task magic is changed and xnshadow_unmap will
never be called. I suspect this is true for all skins, but I do not know
if this could cause a crash.

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Gilles Chanteperdrix]

On 9/7/07, Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote:
> Philippe Gerum wrote:
>  > On Fri, 2007-09-07 at 11:27 +0200, Peter Soetens wrote:
>  > > Just in case you hooked off the long discussion about the issues we
> found from
>  > > Xenomai 2.3.2 on:
>  > >
>  > >   o We are using the xeno_native skin, create Xeno tasks and
> semaphores, but
>  > > have strong indications that the crashes are caused by the memory
> allocation
>  > > scheme of Xenomai in combination with task creation/deletion
>  > >   o We found two ways to break Xenomai, causing a 'Killed'
> (rt_task_delete)
>  > > and causing an OOPS (rt_task_join).
>  > >   o They happen on 2.6.20 and 2.6.22 kernels
>  > >   o On the 2.3 branch, r2429 works, r2433 causes the faults. The patch
> is
>  > > small, and in the ChangLog:
>  > >
>  > > 2007-05-11  Philippe Gerum  <rpm@xenomai.org>
>  > >
>  > >     * include/nucleus/heap.h (xnfreesafe): Use xnpod_current_p() when
>  > >     checking for deferral.
>  > >
>  > >     * include/nucleus/pod.h (xnpod_current_p): Give exec mode
>  > >     awareness to this predicate, checking for primary/secondary mode
>  > >     of shadows.
>  > >
>  > > 2007-05-11  Gilles Chanteperdrix  <gilles.chanteperdrix@xenomai.org>
>  > >
>  > >     * ksrc/skins: Always defer thread memory release in deletion hook
>  > >     by calling xnheap_schedule_free() instead of xnfreesafe().
>  > >
>  > >   o We reverted this patch on HEAD of the 2.3 branch, but got -ENOMEM
> errors
>  > > during Xenomai resource allocations, indicating that later changes
> depend on
>  > > this patch. So we use clean HEAD again further on to find the causes:
>  > >  o A first test (in Orocos) creates one thread, two semaphores, lets it
> wait
>  > > on them and cleans up the thread.
>  >
>  > Please point me at the actual Orocos test code that breaks, with the
>  > hope to get a fairly standalone test case from it; if you do have a
>  > standalone test case already, this would be even better. I intend to
>  > address this issue asap.
>
> Before you have a piece of code that causes the crash, I gave a look at
> the code involved. The only suspicious thing I see is that the correct
> working of native skins thread termination depends on the execution
> order of the two deletion hooks, the one in task.c and the one in
> syscall.c. As a matter of fact, if the one in task.c is executed before
> the one in syscall.c, the task magic is changed and xnshadow_unmap will
> never be called. I suspect this is true for all skins, but I do not know
> if this could cause a crash.

There are two magics involved, this supposition is wrong.

-- 
                                               Gilles Chanteperdrix

Re: [Xenomai-core] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Peter Soetens]

[-- Attachment #1: Type: text/plain, Size: 3520 bytes --]


Sorry for the slow follow ups, I'm nailing this down in my spare time...

On Friday 07 September 2007 14:46:40 Philippe Gerum wrote:
> On Fri, 2007-09-07 at 11:27 +0200, Peter Soetens wrote:
> > Just in case you hooked off the long discussion about the issues we found
> > from Xenomai 2.3.2 on:
> >
> >   o We are using the xeno_native skin, create Xeno tasks and semaphores,
> > but have strong indications that the crashes are caused by the memory
> > allocation scheme of Xenomai in combination with task creation/deletion
> >   o We found two ways to break Xenomai, causing a 'Killed'
> > (rt_task_delete) and causing an OOPS (rt_task_join).
> >   o They happen on 2.6.20 and 2.6.22 kernels
> >   o On the 2.3 branch, r2429 works, r2433 causes the faults. The patch is
> > small, and in the ChangLog:
> >
> > 2007-05-11  Philippe Gerum  <rpm@xenomai.org>
> >
> >     * include/nucleus/heap.h (xnfreesafe): Use xnpod_current_p() when
> >     checking for deferral.
> >
> >     * include/nucleus/pod.h (xnpod_current_p): Give exec mode
> >     awareness to this predicate, checking for primary/secondary mode
> >     of shadows.
> >
> > 2007-05-11  Gilles Chanteperdrix  <gilles.chanteperdrix@xenomai.org>
> >
> >     * ksrc/skins: Always defer thread memory release in deletion hook
> >     by calling xnheap_schedule_free() instead of xnfreesafe().
> >
> >   o We reverted this patch on HEAD of the 2.3 branch, but got -ENOMEM
> > errors during Xenomai resource allocations, indicating that later changes
> > depend on this patch. So we use clean HEAD again further on to find the
> > causes: o A first test (in Orocos) creates one thread, two semaphores,
> > lets it wait on them and cleans up the thread.
>
> Please point me at the actual Orocos test code that breaks, with the
> hope to get a fairly standalone test case from it; if you do have a
> standalone test case already, this would be even better. I intend to
> address this issue asap.

I stripped the OS layer of Orocos for xenomai and built a testcase, which 
causes a complete lockup with that. In order to avoid the lock-up, comment 
some thread/mutex/semaphore creations. All these classes call the functions 
from fosi.c and fosi_internal_join.cpp. So the testcase could be reduced to 
call only these functions...
It seems you need to construct a number of threads and mutexes in the 
same application before it happens. With only 'little' use of these 
primitives, our applications run fine 1000's times in a row.

>
> >  o During rt_task_delete, our program gets 'Killed' (without joinable
> > thread), hence a user space problem. However, gdb is of no use, all
> > thread info is lost.
>
> SIGKILL is sent from the nucleus upon a call to rt_task_delete() which
> targets a non-current task, in order to make sure this user-space task

I don't understand what a non-current task is. However, the Kill indeed came 
from not using rt_task_join().  So I consider this issue as an Orocos bug, 
although previously, we did not get this Kill signal though. Now remains the 
oopses...

> [ oops report ] ...
>
> The only explanation looking at this backtrace is that the system heap
> has been corrupted by the previous exit; likely a side effect of the
> deferral.

We get oopses as well when using the rt_task_join construct. It seems that if 
in an application we use 'to much primitives', it corrupts the heap. This 
would also explain why that specific Xenomai patch causes our problems.

Peter
-- 
Peter Soetens -- FMTC -- <http://www.fmtc.be>

[-- Attachment #2: rtt-test.tgz --]
[-- Type: application/x-tgz, Size: 42411 bytes --]

Re: [Xenomai-core] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Peter Soetens]

Quoting Peter Soetens <peter.soetens@domain.hid>:
> I stripped the OS layer of Orocos for xenomai and built a testcase, which
> causes a complete lockup with that. In order to avoid the lock-up, comment
> some thread/mutex/semaphore creations.

that's in trigger-bug.cpp, use 'make' to create the executable.

Peter

-- 
www.fmtc.be

Re: [Xenomai-core] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Philippe Gerum]

On Thu, 2007-09-13 at 10:06 +0200, Peter Soetens wrote:
> >
> > Please point me at the actual Orocos test code that breaks, with the
> > hope to get a fairly standalone test case from it; if you do have a
> > standalone test case already, this would be even better. I intend to
> > address this issue asap.
> 
> I stripped the OS layer of Orocos for xenomai and built a testcase, which 
> causes a complete lockup with that. In order to avoid the lock-up, comment 
> some thread/mutex/semaphore creations. All these classes call the functions 
> from fosi.c and fosi_internal_join.cpp. So the testcase could be reduced to 
> call only these functions...
> It seems you need to construct a number of threads and mutexes in the 
> same application before it happens. With only 'little' use of these 
> primitives, our applications run fine 1000's times in a row.

Ok. I can confirm that something tragically bugous is happening at
nucleus level when running this testcase. So far, I already triggered
the -ENOMEM issue, and a nifty crash during task creation. I'll dig
this, thanks.

-- 
Philippe.

Re: [Xenomai-core] [FIX] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Philippe Gerum]

On Fri, 2007-09-07 at 11:27 +0200, Peter Soetens wrote:
> Just in case you hooked off the long discussion about the issues we found from
> Xenomai 2.3.2 on:
> 
>   o We are using the xeno_native skin, create Xeno tasks and semaphores, but 
> have strong indications that the crashes are caused by the memory allocation 
> scheme of Xenomai in combination with task creation/deletion
>   o We found two ways to break Xenomai, causing a 'Killed' (rt_task_delete) 
> and causing an OOPS (rt_task_join).
>   o They happen on 2.6.20 and 2.6.22 kernels
>   o On the 2.3 branch, r2429 works, r2433 causes the faults. The patch is 
> small, and in the ChangLog: 
> 

Please try this patch against v2.3.x. A double free issue on a task TCB
already scheduled for memory release was causing all sorts of troubles,
basically trashing the system heap afterwards:

Index: ChangeLog
===================================================================
--- ChangeLog	(revision 2990)
+++ ChangeLog	(revision 2992)
@@ -1,3 +1,19 @@
+2007-09-15  Philippe Gerum  <rpm@xenomai.org>
+
+	* ksrc/skins/vrtx/syscall.c (__sc_tecreate): 
+	* ksrc/skins/vxworks/syscall.c (__wind_task_init): 
+	* ksrc/skins/native/syscall.c (__rt_task_create): Protect against
+	duplicate TCB memory release (first issued by the deletion hook)
+	upon task creation error.
+	
+	* ksrc/skins/native/syscall.c (__rt_task_create): Make sure we
+	attempt to release the TCB memory after any failure to shadow the
+	task.
+
+	* ksrc/skins/native/task.c (rt_task_create): Get out faster from
+	the error case by calling xnpod_delete_thread() upon registration
+	failure.
+
 2007-09-04  Johan Borkhuis <j.borkhuis@domain.hid>
 
 	* ksrc/nucleus/pod.c: Make the watchdog timeout value configurable.
Index: ksrc/skins/vrtx/syscall.c
===================================================================
--- ksrc/skins/vrtx/syscall.c	(revision 2990)
+++ ksrc/skins/vrtx/syscall.c	(revision 2992)
@@ -86,6 +86,8 @@
 		goto done;
 	}
 
+	xnthread_clear_state(&task->threadbase, XNZOMBIE);
+
 	tid =
 	    sc_tecreate_inner(task, NULL, tid, prio, mode, 0, 0, NULL, 0, &err);
 
@@ -98,7 +100,7 @@
 		err = xnshadow_map(&task->threadbase, u_completion);
 	}
 
-	if (err)
+	if (err && !xnthread_test_state(&task->threadbase, XNZOMBIE))
 		xnfree(task);
 
       done:
Index: ksrc/skins/vxworks/syscall.c
===================================================================
--- ksrc/skins/vxworks/syscall.c	(revision 2990)
+++ ksrc/skins/vxworks/syscall.c	(revision 2992)
@@ -111,6 +111,8 @@
 		return -ENOMEM;
 	}
 
+	xnthread_clear_state(&task->threadbase, XNZOMBIE);
+
 	/* Force FPU support in user-space. This will lead to a no-op if
 	   the platform does not support it. */
 
@@ -132,7 +134,7 @@
 			xnshadow_signal_completion(u_completion, err);
 	}
 
-	if (err)
+	if (err && !xnthread_test_state(&task->threadbase, XNZOMBIE))
 		xnfree(task);
 
 	return err;
Index: ksrc/skins/native/task.c
===================================================================
--- ksrc/skins/native/task.c	(revision 2990)
+++ ksrc/skins/native/task.c	(revision 2992)
@@ -286,7 +286,7 @@
 				       &xnthread_handle(&task->thread_base),
 				       NULL);
 		if (err)
-			rt_task_delete(task);
+			xnpod_delete_thread(&task->thread_base);
 		else if (!*name)
 			/* /proc/xenomai/sched will dump no name for the anonymous
 			   task, but the registry still has a stable reference
Index: ksrc/skins/native/syscall.c
===================================================================
--- ksrc/skins/native/syscall.c	(revision 2990)
+++ ksrc/skins/native/syscall.c	(revision 2992)
@@ -125,8 +125,8 @@
 	char name[XNOBJECT_NAME_LEN];
 	struct rt_arg_bulk bulk;
 	RT_TASK_PLACEHOLDER ph;
+	RT_TASK *task = NULL;
 	int err, prio, mode;
-	RT_TASK *task;
 
 	/* Completion descriptor our parent thread is pending on -- may be NULL. */
 	u_completion = (xncompletion_t __user *)__xn_reg_arg2(regs);
@@ -170,6 +170,8 @@
 		goto fail;
 	}
 
+	xnthread_clear_state(&task->thread_base, XNZOMBIE);
+
 	/* Force FPU support in user-space. This will lead to a no-op if
 	   the platform does not support it. */
 
@@ -183,13 +185,21 @@
 				  sizeof(ph));
 		err = xnshadow_map(&task->thread_base, u_completion);
 	} else {
-		xnfree(task);
 		/* Unblock and pass back error code. */
 fail:
 		if (u_completion)
 			xnshadow_signal_completion(u_completion, err);
 	}
 
+	/* Task memory could have been released by an indirect call to
+	 * the deletion hook, after xnpod_delete_thread() has been
+	 * issued from rt_task_create() (e.g. upon registration
+	 * error). We avoid double memory release when the XNZOMBIE
+	 * flag is raised, meaning the deletion hook has run, and the
+	 * TCB memory is already scheduled for release. */
+	if (err && task != NULL && !xnthread_test_state(&task->thread_base, XNZOMBIE))
+		xnfree(task);
+
 	return err;
 }
 

-- 
Philippe.

Re: [Xenomai-core] [FIX] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Gilles Chanteperdrix]

Philippe Gerum wrote:
 > 
 > Please try this patch against v2.3.x. A double free issue on a task TCB
 > already scheduled for memory release was causing all sorts of troubles,
 > basically trashing the system heap afterwards:

Is not there anything similar needed for the posix skin ?

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] [FIX] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Philippe Gerum]

On Sun, 2007-09-16 at 19:49 +0200, Gilles Chanteperdrix wrote:
> Philippe Gerum wrote:
>  > 
>  > Please try this patch against v2.3.x. A double free issue on a task TCB
>  > already scheduled for memory release was causing all sorts of troubles,
>  > basically trashing the system heap afterwards:
> 
> Is not there anything similar needed for the posix skin ?
> 

pthread_create() does allocate the TCB itself, and properly releases it
upon error while creating the thread, so the syscall wrapper does not
have to free any TCB memory by itself, which is a prerequisite to enter
the double release situation. There is no possibility to run the
deletion hook indirectly on behalf of pthread_create() either, so I
think it's safe.

-- 
Philippe.

Re: [Xenomai-core] [FIX] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Peter Soetens]

On Saturday 15 September 2007 20:52:59 Philippe Gerum wrote:
> On Fri, 2007-09-07 at 11:27 +0200, Peter Soetens wrote:
> > Just in case you hooked off the long discussion about the issues we found
> > from Xenomai 2.3.2 on:
> >
> >   o We are using the xeno_native skin, create Xeno tasks and semaphores,
> > but have strong indications that the crashes are caused by the memory
> > allocation scheme of Xenomai in combination with task creation/deletion
> >   o We found two ways to break Xenomai, causing a 'Killed'
> > (rt_task_delete) and causing an OOPS (rt_task_join).
> >   o They happen on 2.6.20 and 2.6.22 kernels
> >   o On the 2.3 branch, r2429 works, r2433 causes the faults. The patch is
> > small, and in the ChangLog:
>
> Please try this patch against v2.3.x. A double free issue on a task TCB
> already scheduled for memory release was causing all sorts of troubles,
> basically trashing the system heap afterwards:

Thanks, we'll try, test and report ASAP (= somewhere this week.)

Peter
-- 
Peter Soetens -- FMTC -- <http://www.fmtc.be>

Re: [Xenomai-core] [FIX] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Peter Soetens]

On Saturday 15 September 2007 20:52:59 Philippe Gerum wrote:
> On Fri, 2007-09-07 at 11:27 +0200, Peter Soetens wrote:
> > Just in case you hooked off the long discussion about the issues we found
> > from Xenomai 2.3.2 on:
> >
> >   o We are using the xeno_native skin, create Xeno tasks and semaphores,
> > but have strong indications that the crashes are caused by the memory
> > allocation scheme of Xenomai in combination with task creation/deletion
> >   o We found two ways to break Xenomai, causing a 'Killed'
> > (rt_task_delete) and causing an OOPS (rt_task_join).
> >   o They happen on 2.6.20 and 2.6.22 kernels
> >   o On the 2.3 branch, r2429 works, r2433 causes the faults. The patch is
> > small, and in the ChangLog:
>
> Please try this patch against v2.3.x. A double free issue on a task TCB
> already scheduled for memory release was causing all sorts of troubles,
> basically trashing the system heap afterwards:

Andy just confirmed that this patch fixes his issues !

Thanks a lot,
Peter

-- 
Peter Soetens -- FMTC -- <http://www.fmtc.be>