[Xenomai-help] General question on Native Skin tasks

[Xenomai-help] General question on Native Skin tasks

[Brian L.]

If I create a native-skin RT_TASK from userspace with no flags, i.e.

void task(void*)
{
    for (;;) ;
}
int main()
{
   RT_TASK t;
   rt_task_create(&t, 0, 3, 0);
   rt_task_start(&t,task,0);
   (do something which blocks)
}

Should that task starve all other tasks? In what ways is it different
behaviorally from an ordinary posix thread? I ask because I have a
thread that might be spinning and a frozen system (back in the rtlinux
days, spinning in the real-time domain was a surefire way to freeze
without any other sign of trouble). If it is the case that this task
outweighs an ordinary linux thread/task, how can I make it the same?

Also,  I haven't completely tracked this down yet, but xenomai seems
to be page-faulting in a loop and exploding rather spectacularly in a
native-skin multithreaded program that doesn't do anything outside of
relatively ordinary queue/mutex/task stuff. I'm upgrading to trunk to
see if it goes away.

Short of a serial cable, is there a way to capture Oops/Panic text as
it flies by?

Re: [Xenomai-help] General question on Native Skin tasks

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1752 bytes --]

Brian L. wrote:
> If I create a native-skin RT_TASK from userspace with no flags, i.e.
> 
> void task(void*)
> {
>     for (;;) ;
> }
> int main()
> {
>    RT_TASK t;
>    rt_task_create(&t, 0, 3, 0);
>    rt_task_start(&t,task,0);
>    (do something which blocks)
> }

mlockall left out for simplicity? Or is it also missing on your real
test? In the latter case, occasional application crashes are "normal"
(as described below).

Philippe, you suggested some code for detecting this. We should really,
really add this soon (maybe to the exception path)!

> 
> Should that task starve all other tasks? In what ways is it different
> behaviorally from an ordinary posix thread? I ask because I have a
> thread that might be spinning and a frozen system (back in the rtlinux
> days, spinning in the real-time domain was a surefire way to freeze
> without any other sign of trouble). If it is the case that this task
> outweighs an ordinary linux thread/task, how can I make it the same?

A task that has high priority than other system tasks has to outweigh
them. This has nothing to to with POSIX (standard Linux included!),
RTLinux, Xenomai, or whatever. But Xenomai has a simple watchdog to
recover from run-away RT threads.

> 
> Also,  I haven't completely tracked this down yet, but xenomai seems
> to be page-faulting in a loop and exploding rather spectacularly in a
> native-skin multithreaded program that doesn't do anything outside of
> relatively ordinary queue/mutex/task stuff. I'm upgrading to trunk to
> see if it goes away.
> 
> Short of a serial cable, is there a way to capture Oops/Panic text as
> it flies by?

linux/Documentation/networking/netconsole.txt is another option.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [Xenomai-help] General question on Native Skin tasks

[Philippe Gerum]

Jan Kiszka wrote:
> Brian L. wrote:
> 
>>If I create a native-skin RT_TASK from userspace with no flags, i.e.
>>
>>void task(void*)
>>{
>>    for (;;) ;
>>}
>>int main()
>>{
>>   RT_TASK t;
>>   rt_task_create(&t, 0, 3, 0);
>>   rt_task_start(&t,task,0);
>>   (do something which blocks)
>>}
> 
> 
> mlockall left out for simplicity? Or is it also missing on your real
> test? In the latter case, occasional application crashes are "normal"
> (as described below).
> 
> Philippe, you suggested some code for detecting this. We should really,
> really add this soon (maybe to the exception path)!
> 

The submitted patch works pretty well detecting unlocked memory, I'm 
using it right now, but I'd like something a bit more self-explanatory 
than just receiving SIGXCPU. I don't think the execption path is the 
right place to put this, since the mlockall issue causes random bugs, 
and you likely want to detect them early and unconditionally.

> 
>>Should that task starve all other tasks? In what ways is it different
>>behaviorally from an ordinary posix thread? I ask because I have a
>>thread that might be spinning and a frozen system (back in the rtlinux
>>days, spinning in the real-time domain was a surefire way to freeze
>>without any other sign of trouble). If it is the case that this task
>>outweighs an ordinary linux thread/task, how can I make it the same?
> 
> 
> A task that has high priority than other system tasks has to outweigh
> them. This has nothing to to with POSIX (standard Linux included!),
> RTLinux, Xenomai, or whatever. But Xenomai has a simple watchdog to
> recover from run-away RT threads.
> 
> 
>>Also,  I haven't completely tracked this down yet, but xenomai seems
>>to be page-faulting in a loop and exploding rather spectacularly in a
>>native-skin multithreaded program that doesn't do anything outside of
>>relatively ordinary queue/mutex/task stuff. I'm upgrading to trunk to
>>see if it goes away.
>>
>>Short of a serial cable, is there a way to capture Oops/Panic text as
>>it flies by?
> 
> 
> linux/Documentation/networking/netconsole.txt is another option.
> 
> Jan
> 


-- 

Philippe.

Re: [Xenomai-help] General question on Native Skin tasks

[Philippe Gerum]

Philippe Gerum wrote:
> Jan Kiszka wrote:
> 
>> Brian L. wrote:
>>
>>> If I create a native-skin RT_TASK from userspace with no flags, i.e.
>>>
>>> void task(void*)
>>> {
>>>    for (;;) ;
>>> }
>>> int main()
>>> {
>>>   RT_TASK t;
>>>   rt_task_create(&t, 0, 3, 0);
>>>   rt_task_start(&t,task,0);
>>>   (do something which blocks)
>>> }
>>
>>
>>
>> mlockall left out for simplicity? Or is it also missing on your real
>> test? In the latter case, occasional application crashes are "normal"
>> (as described below).
>>
>> Philippe, you suggested some code for detecting this. We should really,
>> really add this soon (maybe to the exception path)!
>>
> 
> The submitted patch works pretty well detecting unlocked memory, I'm 
> using it right now, but I'd like something a bit more self-explanatory 
> than just receiving SIGXCPU. I don't think the execption path is the 
> right place to put this, since the mlockall issue causes random bugs, 
> and you likely want to detect them early and unconditionally.
> 

Commit #941 should provide a reliable guard against lack of process 
memory locking.

-- 

Philippe.

Re: [Xenomai-help] General question on Native Skin tasks

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1397 bytes --]

Philippe Gerum wrote:
> Philippe Gerum wrote:
>> Jan Kiszka wrote:
>>
>>> Brian L. wrote:
>>>
>>>> If I create a native-skin RT_TASK from userspace with no flags, i.e.
>>>>
>>>> void task(void*)
>>>> {
>>>>    for (;;) ;
>>>> }
>>>> int main()
>>>> {
>>>>   RT_TASK t;
>>>>   rt_task_create(&t, 0, 3, 0);
>>>>   rt_task_start(&t,task,0);
>>>>   (do something which blocks)
>>>> }
>>>
>>>
>>>
>>> mlockall left out for simplicity? Or is it also missing on your real
>>> test? In the latter case, occasional application crashes are "normal"
>>> (as described below).
>>>
>>> Philippe, you suggested some code for detecting this. We should really,
>>> really add this soon (maybe to the exception path)!
>>>
>>
>> The submitted patch works pretty well detecting unlocked memory, I'm
>> using it right now, but I'd like something a bit more self-explanatory
>> than just receiving SIGXCPU. I don't think the execption path is the
>> right place to put this, since the mlockall issue causes random bugs,
>> and you likely want to detect them early and unconditionally.
>>
> 
> Commit #941 should provide a reliable guard against lack of process
> memory locking.
> 

Hmm, a simple test using the latency tool with disabled mlockall did not
yet show any effect on my system. Shouldn't there pop up some message
when starting such a "broken" program?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [Xenomai-help] General question on Native Skin tasks

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]

Jan Kiszka wrote:
> Philippe Gerum wrote:
>> Philippe Gerum wrote:
>>> Jan Kiszka wrote:
>>>
>>>> Brian L. wrote:
>>>>
>>>>> If I create a native-skin RT_TASK from userspace with no flags, i.e.
>>>>>
>>>>> void task(void*)
>>>>> {
>>>>>    for (;;) ;
>>>>> }
>>>>> int main()
>>>>> {
>>>>>   RT_TASK t;
>>>>>   rt_task_create(&t, 0, 3, 0);
>>>>>   rt_task_start(&t,task,0);
>>>>>   (do something which blocks)
>>>>> }
>>>>
>>>>
>>>> mlockall left out for simplicity? Or is it also missing on your real
>>>> test? In the latter case, occasional application crashes are "normal"
>>>> (as described below).
>>>>
>>>> Philippe, you suggested some code for detecting this. We should really,
>>>> really add this soon (maybe to the exception path)!
>>>>
>>> The submitted patch works pretty well detecting unlocked memory, I'm
>>> using it right now, but I'd like something a bit more self-explanatory
>>> than just receiving SIGXCPU. I don't think the execption path is the
>>> right place to put this, since the mlockall issue causes random bugs,
>>> and you likely want to detect them early and unconditionally.
>>>
>> Commit #941 should provide a reliable guard against lack of process
>> memory locking.
>>
> 
> Hmm, a simple test using the latency tool with disabled mlockall did not
> yet show any effect on my system. Shouldn't there pop up some message
> when starting such a "broken" program?
> 

The usual "rebuild your stuff before you try" (I missed a kernel
rebuild). Works very nicely!

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [Xenomai-help] General question on Native Skin tasks

[Brian L.]

Ok; What xenomai priority in the 1-99 scheme corresponds to the
priority of normal user-level posix threads?

The application I'm building has realtime and non-realtime components
which share some native-skin objects. I am constructing my
time-critical and non-time-critical threads using rt_task_create. I
would like almost all of them to behave exactly as ordinary
non-xenomai posix threads do with regard to priority and starvation.

The example was contrived to support my question. In practice, I do
call mlockall. Also, my development machine has no swap space, so
mlockall probably shouldn't have an effect.

What does this watchdog do and where is it documented? I'm surprised I
haven't heard it mentioned before, since it seems rather important.

So far, I've not intended to introduce any "real-time" features; What
I'm doing is very vanilla stuff that I'd normally do with pthreads.

Brian


On 4/17/06, Jan Kiszka <jan.kiszka@domain.hid> wrote:
> Brian L. wrote:
> > If I create a native-skin RT_TASK from userspace with no flags, i.e.
> >
> > void task(void*)
> > {
> >     for (;;) ;
> > }
> > int main()
> > {
> >    RT_TASK t;
> >    rt_task_create(&t, 0, 3, 0);
> >    rt_task_start(&t,task,0);
> >    (do something which blocks)
> > }
>
> mlockall left out for simplicity? Or is it also missing on your real
> test? In the latter case, occasional application crashes are "normal"
> (as described below).
>
> Philippe, you suggested some code for detecting this. We should really,
> really add this soon (maybe to the exception path)!
>
> >
> > Should that task starve all other tasks? In what ways is it different
> > behaviorally from an ordinary posix thread? I ask because I have a
> > thread that might be spinning and a frozen system (back in the rtlinux
> > days, spinning in the real-time domain was a surefire way to freeze
> > without any other sign of trouble). If it is the case that this task
> > outweighs an ordinary linux thread/task, how can I make it the same?
>
> A task that has high priority than other system tasks has to outweigh
> them. This has nothing to to with POSIX (standard Linux included!),
> RTLinux, Xenomai, or whatever. But Xenomai has a simple watchdog to
> recover from run-away RT threads.
>
> >
> > Also,  I haven't completely tracked this down yet, but xenomai seems
> > to be page-faulting in a loop and exploding rather spectacularly in a
> > native-skin multithreaded program that doesn't do anything outside of
> > relatively ordinary queue/mutex/task stuff. I'm upgrading to trunk to
> > see if it goes away.
> >
> > Short of a serial cable, is there a way to capture Oops/Panic text as
> > it flies by?
>
> linux/Documentation/networking/netconsole.txt is another option.
>
> Jan
>
>
>
>

Re: [Xenomai-help] General question on Native Skin tasks

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2309 bytes --]

Brian L. wrote:
> Ok; What xenomai priority in the 1-99 scheme corresponds to the
> priority of normal user-level posix threads?

Depends on the mode of your xenomai thread: when in primary mode (the
hard-rt mode where the xeno threads live by default), those 99 levels
are actually above all linux priorities.

But when your thread enters secondary mode to issue some linux syscall
or handle a normal signal, its priority gets mapped 1:1 on the linux
range. This means, e.g., that a xenomai thread with prio 50 in secondary
mode gets overruled by a normal linux thread with prio 60 but not by a
xenomai thread in primary mode even with prio 1.

> 
> The application I'm building has realtime and non-realtime components
> which share some native-skin objects. I am constructing my
> time-critical and non-time-critical threads using rt_task_create. I
> would like almost all of them to behave exactly as ordinary
> non-xenomai posix threads do with regard to priority and starvation.

Why do you create the non-rt threads also via rt_task_create? This just
introduces unneeded load on the RT subsystem.

> 
> The example was contrived to support my question. In practice, I do
> call mlockall. Also, my development machine has no swap space, so
> mlockall probably shouldn't have an effect.

I does, thanks to the lacy memory allocation scheme of glibc (alloc on
access).

> 
> What does this watchdog do and where is it documented? I'm surprised I
> haven't heard it mentioned before, since it seems rather important.

It's a kernel option (isn't it mentioned somewhere in the docs? then it
needs fixing).

The scheme is that after 4 seconds (if I remember the number correctly)
of continuous xenomai activity with linux leaving no breath, the
currently running xenomai thread gets suspended. This may be the wrong
one when you are unlucky, but then one of the next round will hit the
offender. We may think about improving this scheme a bit (suspending the
lowest-prio thread?). I helps to return your system to "interactive
mode", analysing what went wrong and maybe cleaning it up.

> 
> So far, I've not intended to introduce any "real-time" features; What
> I'm doing is very vanilla stuff that I'd normally do with pthreads.
> 
> Brian
> 

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] General question on Native Skin tasks

[Brian L.]

> Why do you create the non-rt threads also via rt_task_create? This just
> introduces unneeded load on the RT subsystem.

I was under the impression that to interact with xenomai objects
(RT_QUEUE, RT_MUTEX, etc), I needed to be in a xenomai thread. I
assumed that because every mutex implementation I've seen has relied
heavily on the underlying thread library (i.e. to sleep or yield) and
mixing thread libraries seemed like a bad idea.

So, is this what you're suggesting:

- Link against pthreads AND native
- Start non-realtime threads using pthread_create and real-time
threads using rt_task_create/rt_task_start
- use RT_MUTEX's/RT_QUEUE's for situations in which there is
contention/communication among realtime and nonrealtime threads
- use pthreads mutexes/queues for situations of
contention/communication among non-realtime threads

It sounds workable to me, assuming the queues/mutexes/etc. don't stop
working when both threading libraries are involved.

Re: [Xenomai-help] General question on Native Skin tasks

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1683 bytes --]

Brian L. wrote:
>> Why do you create the non-rt threads also via rt_task_create? This just
>> introduces unneeded load on the RT subsystem.
> 
> I was under the impression that to interact with xenomai objects
> (RT_QUEUE, RT_MUTEX, etc), I needed to be in a xenomai thread. I
> assumed that because every mutex implementation I've seen has relied
> heavily on the underlying thread library (i.e. to sleep or yield) and
> mixing thread libraries seemed like a bad idea.
> 
> So, is this what you're suggesting:
> 
> - Link against pthreads AND native

By linking against native, you implicitly also link against pthread
(native threads are based on pthreads).

> - Start non-realtime threads using pthread_create and real-time
> threads using rt_task_create/rt_task_start

Yep.

> - use RT_MUTEX's/RT_QUEUE's for situations in which there is
> contention/communication among realtime and nonrealtime threads

Yes, but when blocking native services are invoked, the caller has to be
a native thread again. Only pure non-RT threads with no shared critical
paths can be plain pthreads. When in doubt, study the native
documentation regarding allowed contexts. The statement "switches to
primary mode" indicates that a native thread is required.

> - use pthreads mutexes/queues for situations of
> contention/communication among non-realtime threads

That's totally fine and also improve the efficiency (something that has
to be sacrificed for determinism sometimes).

> 
> It sounds workable to me, assuming the queues/mutexes/etc. don't stop
> working when both threading libraries are involved.

Hope I could clarify the situation.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] General question on Native Skin tasks

[Brian L.]

> Yes, but when blocking native services are invoked, the caller has to be
> a native thread again. Only pure non-RT threads with no shared critical
> paths can be plain pthreads. When in doubt, study the native
> documentation regarding allowed contexts. The statement "switches to
> primary mode" indicates that a native thread is required.

So in this case, can I still call the functions from a thread created
using pthread_create (assuming that they may momentarily increase in
priority)?

Can you explain "no shared critical paths"? Would this include
contending for a mutex-protected resource which is also used by a
real-time thread?

All of my non-time-critical threads are doing some form of
communication with real-time threads, either through a queue or
through mutex-protected memory. Can they still be pthread_create
threads?

> Hope I could clarify the situation.

You've been supremely helpful. I apologize for asking so many questions.


>
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
>
>
>
>

Re: [Xenomai-help] General question on Native Skin tasks

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1683 bytes --]

Brian L. wrote:
>> Yes, but when blocking native services are invoked, the caller has to be
>> a native thread again. Only pure non-RT threads with no shared critical
>> paths can be plain pthreads. When in doubt, study the native
>> documentation regarding allowed contexts. The statement "switches to
>> primary mode" indicates that a native thread is required.
> 
> So in this case, can I still call the functions from a thread created
> using pthread_create (assuming that they may momentarily increase in
> priority)?

You can call native (or more general: Xenomai) services from non-RT
threads when they do not let your thread block (or contain the risk of
blocking). To block on some RT resources like a queue or a mutex you
need a RT context, e.g. a native thread.

> 
> Can you explain "no shared critical paths"? Would this include
> contending for a mutex-protected resource which is also used by a
> real-time thread?

Yes, this is such a critical path. You may want that every thread
holding the mutex does this only for a bounded time. To ensure this,
priority inheritance may be applied on the mutex holder, thus it has to
be a rt-thread.

> 
> All of my non-time-critical threads are doing some form of
> communication with real-time threads, either through a queue or
> through mutex-protected memory. Can they still be pthread_create
> threads?

Nope, in this case they have to be extended, i.e. created as native threads.

> 
>> Hope I could clarify the situation.
> 
> You've been supremely helpful. I apologize for asking so many questions.
> 

No problem, I always hope this helps others implicitly as well. :)

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [Xenomai-help] General question on Native Skin tasks

[Li Yi]

On Wed, 2006-04-19 at 21:45 +0200, Jan Kiszka wrote:

> > 
> >> Hope I could clarify the situation.
> > 
> > You've been supremely helpful. I apologize for asking so many questions.
> > 
> 
> No problem, I always hope this helps others implicitly as well. :)
> 
> Jan
> 

Exactly. Thanks Jan. This thread is very helpful.

> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help