Re: [Xenomai-core] Summary: Xenomai 2.3.2 and 2.4 lock-ups and OOPSes

[Peter Soetens <peter.soetens@domain.hid>]

[-- Attachment #1: Type: text/plain, Size: 3520 bytes --]


Sorry for the slow follow ups, I'm nailing this down in my spare time...

On Friday 07 September 2007 14:46:40 Philippe Gerum wrote:
> On Fri, 2007-09-07 at 11:27 +0200, Peter Soetens wrote:
> > Just in case you hooked off the long discussion about the issues we found
> > from Xenomai 2.3.2 on:
> >
> >   o We are using the xeno_native skin, create Xeno tasks and semaphores,
> > but have strong indications that the crashes are caused by the memory
> > allocation scheme of Xenomai in combination with task creation/deletion
> >   o We found two ways to break Xenomai, causing a 'Killed'
> > (rt_task_delete) and causing an OOPS (rt_task_join).
> >   o They happen on 2.6.20 and 2.6.22 kernels
> >   o On the 2.3 branch, r2429 works, r2433 causes the faults. The patch is
> > small, and in the ChangLog:
> >
> > 2007-05-11  Philippe Gerum  <rpm@xenomai.org>
> >
> >     * include/nucleus/heap.h (xnfreesafe): Use xnpod_current_p() when
> >     checking for deferral.
> >
> >     * include/nucleus/pod.h (xnpod_current_p): Give exec mode
> >     awareness to this predicate, checking for primary/secondary mode
> >     of shadows.
> >
> > 2007-05-11  Gilles Chanteperdrix  <gilles.chanteperdrix@xenomai.org>
> >
> >     * ksrc/skins: Always defer thread memory release in deletion hook
> >     by calling xnheap_schedule_free() instead of xnfreesafe().
> >
> >   o We reverted this patch on HEAD of the 2.3 branch, but got -ENOMEM
> > errors during Xenomai resource allocations, indicating that later changes
> > depend on this patch. So we use clean HEAD again further on to find the
> > causes: o A first test (in Orocos) creates one thread, two semaphores,
> > lets it wait on them and cleans up the thread.
>
> Please point me at the actual Orocos test code that breaks, with the
> hope to get a fairly standalone test case from it; if you do have a
> standalone test case already, this would be even better. I intend to
> address this issue asap.

I stripped the OS layer of Orocos for xenomai and built a testcase, which 
causes a complete lockup with that. In order to avoid the lock-up, comment 
some thread/mutex/semaphore creations. All these classes call the functions 
from fosi.c and fosi_internal_join.cpp. So the testcase could be reduced to 
call only these functions...
It seems you need to construct a number of threads and mutexes in the 
same application before it happens. With only 'little' use of these 
primitives, our applications run fine 1000's times in a row.

>
> >  o During rt_task_delete, our program gets 'Killed' (without joinable
> > thread), hence a user space problem. However, gdb is of no use, all
> > thread info is lost.
>
> SIGKILL is sent from the nucleus upon a call to rt_task_delete() which
> targets a non-current task, in order to make sure this user-space task

I don't understand what a non-current task is. However, the Kill indeed came 
from not using rt_task_join().  So I consider this issue as an Orocos bug, 
although previously, we did not get this Kill signal though. Now remains the 
oopses...

> [ oops report ] ...
>
> The only explanation looking at this backtrace is that the system heap
> has been corrupted by the previous exit; likely a side effect of the
> deferral.

We get oopses as well when using the rt_task_join construct. It seems that if 
in an application we use 'to much primitives', it corrupts the heap. This 
would also explain why that specific Xenomai patch causes our problems.

Peter
-- 
Peter Soetens -- FMTC -- <http://www.fmtc.be>

[-- Attachment #2: rtt-test.tgz --]
[-- Type: application/x-tgz, Size: 42411 bytes --]