[Xenomai] mutate/start RT_TASK a posix thread?

[Xenomai] mutate/start RT_TASK a posix thread?

[Michael Haberler]

we're happily using RT threads using the Xenomai 2 thread API

is it possible _using this API_ to create/mutate/relax such a thread intentionally into a Posix thread but retaining the API usage?

I can do any conditional API usage through thread parameters to avoid calls which will not work for a relaxed thread, but I would like to retain the API (I guess I could switch to the posix skin but I would like to avoid another learning curve/test cycle for now)

going forward/Xenomai 3: what would be your recommendation to address the issue long-term? 




(may sound like whacky question but great use case around here - turns out many jobs do not need RT capabilities but it would be handy to retain the flow)

thanks in advance,

- Michael


API used from http://www.xenomai.org/documentation/trunk/html/api/group__task.html :

rt_task_create
rt_task_delete
rt_task_inquire
rt_task_join
rt_task_resume
rt_task_self
rt_task_set_affinity
rt_task_set_mode
rt_task_set_periodic
rt_task_start,
rt_task_suspend
rt_task_unblock
rt_task_wait_period

ps: update on rtdm_native: 'sortof works' - I ran into a problem with unbalanced IRQ's which seems unrelated to rtdm_native but more related to my kernel fu; with a determined push/more clue it should be possible to transpose into 'fully works'

Re: [Xenomai] mutate/start RT_TASK a posix thread?

[Philippe Gerum]

On 07/30/2015 07:26 AM, Michael Haberler wrote:
> we're happily using RT threads using the Xenomai 2 thread API
> 
> is it possible _using this API_ to create/mutate/relax such a thread intentionally into a Posix thread but retaining the API usage?

It is possible to run them as low priority Xenomai threads, assigning
them to the SCHED_OTHER class. The native API is retained for those
threads, but their main runtime mode is relaxed, i.e. the co-kernel
makes sure to switch them back to relaxed mode automatically before
returning to user-space from a syscall which required a switch to
primary mode.

> 
> I can do any conditional API usage through thread parameters to avoid calls which will not work for a relaxed thread, but I would like to retain the API (I guess I could switch to the posix skin but I would like to avoid another learning curve/test cycle for now)
> 
> going forward/Xenomai 3: what would be your recommendation to address the issue long-term? 
>

Xenomai 3 in dual kernel mode extends the feature above, with the
SCHED_WEAK class:
http://xenomai.org/migrating-from-xenomai-2-x-to-3-x/#Scheduling

> 
> 
> 
> (may sound like whacky question but great use case around here - turns out many jobs do not need RT capabilities but it would be handy to retain the flow)
> 

If I interpreted your question properly, it's definitely a legitimate
use case; sometimes the non-rt stuff may want to interact with the rt
world using Xenomai APIs, but without getting in the way priority-wise.

-- 
Philippe.

Re: [Xenomai] mutate/start RT_TASK a posix thread?

[Philippe Gerum]

On 07/30/2015 06:25 PM, Philippe Gerum wrote:
> On 07/30/2015 07:26 AM, Michael Haberler wrote:
>> we're happily using RT threads using the Xenomai 2 thread API
>>
>> is it possible _using this API_ to create/mutate/relax such a thread intentionally into a Posix thread but retaining the API usage?
> 
> It is possible to run them as low priority Xenomai threads, assigning
> them to the SCHED_OTHER class. The native API is retained for those

... which is obtained by passing prio=0 to rt_task_create() or
rt_task_shadow().

-- 
Philippe.

Re: [Xenomai] mutate/start RT_TASK a posix thread?

[Michael Haberler]

Philippe,


> Am 30.07.2015 um 18:25 schrieb Philippe Gerum <rpm@xenomai.org>:
> 
> On 07/30/2015 07:26 AM, Michael Haberler wrote:
>> we're happily using RT threads using the Xenomai 2 thread API
>> 
>> is it possible _using this API_ to create/mutate/relax such a thread intentionally into a Posix thread but retaining the API usage?
> 
> It is possible to run them as low priority Xenomai threads, assigning
> them to the SCHED_OTHER class. The native API is retained for those
> threads, but their main runtime mode is relaxed, i.e. the co-kernel
> makes sure to switch them back to relaxed mode automatically before
> returning to user-space from a syscall which required a switch to
> primary mode.

that would give us 'low-priority RT (SCHED_OTHER)' threads which is already a great improvement for us, and it would be simple to do as you outlined in the followup mail


taking things one step further (I'm unsure about the precise semantics of 'relaxed mode', so far I understood it as 'behaves like a normal Linux thread'):

would such a SCHED_OTHER-class thread be able to use _any_ system calls like a normal Linux thread? like file I/O, sockets, poll, etc?

if so - are there any side effects I should be aware of?

(I guess my question really is: does 'relaxed mode equals Linux thread semantics' hold?)


> 
>> 
>> I can do any conditional API usage through thread parameters to avoid calls which will not work for a relaxed thread, but I would like to retain the API (I guess I could switch to the posix skin but I would like to avoid another learning curve/test cycle for now)
>> 
>> going forward/Xenomai 3: what would be your recommendation to address the issue long-term? 
>> 
> 
> Xenomai 3 in dual kernel mode extends the feature above, with the
> SCHED_WEAK class:
> http://xenomai.org/migrating-from-xenomai-2-x-to-3-x/#Scheduling
> 
>> 
>> 
>> 
>> (may sound like whacky question but great use case around here - turns out many jobs do not need RT capabilities but it would be handy to retain the flow)
>> 
> 
> If I interpreted your question properly, it's definitely a legitimate
> use case; sometimes the non-rt stuff may want to interact with the rt
> world using Xenomai APIs, but without getting in the way priority-wise.

exactly. The unknown for me is: can that quasi-non-rt thread do more than a bona-fide RT thread. 

For comparison: in the RT-PREEMPT flavor of our stack, to achieve the same effect we'd just skip the sched_setscheduler(0, SCHED_FIFO, ..) step on thread creation and have the desired result.

thanks for clarifying - that really helps!

Michael


> 
> -- 
> Philippe.

Re: [Xenomai] mutate/start RT_TASK a posix thread?

[Philippe Gerum]

On 07/31/2015 08:50 AM, Michael Haberler wrote:
> Philippe,
> 
> 
>> Am 30.07.2015 um 18:25 schrieb Philippe Gerum <rpm@xenomai.org>:
>>
>> On 07/30/2015 07:26 AM, Michael Haberler wrote:
>>> we're happily using RT threads using the Xenomai 2 thread API
>>>
>>> is it possible _using this API_ to create/mutate/relax such a thread intentionally into a Posix thread but retaining the API usage?
>>
>> It is possible to run them as low priority Xenomai threads, assigning
>> them to the SCHED_OTHER class. The native API is retained for those
>> threads, but their main runtime mode is relaxed, i.e. the co-kernel
>> makes sure to switch them back to relaxed mode automatically before
>> returning to user-space from a syscall which required a switch to
>> primary mode.
> 
> that would give us 'low-priority RT (SCHED_OTHER)' threads which is already a great improvement for us, and it would be simple to do as you outlined in the followup mail
> 
> 
> taking things one step further (I'm unsure about the precise semantics of 'relaxed mode', so far I understood it as 'behaves like a normal Linux thread'):

By relaxed, I mean scheduled by the regular kernel as any other
non-Xenomai thread, not by the Xenomai scheduler. As a consequence of
this, no rt guarantee, no pressure for very short response time.

> 
> would such a SCHED_OTHER-class thread be able to use _any_ system calls like a normal Linux thread? like file I/O, sockets, poll, etc?
> 

Yes. The only issue to care about is not passing file descriptors
obtained by the Xenomai open() call to regular Linux I/O services. The
converse works with the POSIX API though, since the Xenomai I/O
subsystem hands over requests for fds it does not own to the
corresponding glibc call.

> if so - are there any side effects I should be aware of?
> 

None. However, switching a non-rt Xenomai thread back and forth between
the Xenomai and Linux sides involves several scheduling operations each
time (one less for 3.x compared to 2.x though). Therefore, it is better
if the low priority thread does not have to do this at a high rate, it's
faily costly CPU-wise.

> (I guess my question really is: does 'relaxed mode equals Linux thread semantics' hold?)
> 
> 

Almost, with the added bonus of being able to synchronize with other
Xenomai threads (rt and non-rt) as well, which a regular Linux thread can't.

The gist of the matter is about being able to block/sleep on Xenomai
synchronization objects, such as sem4s, mutexes, queues, events, etc: to
do that, the caller has to provide a Xenomai-originated task control
block to the co-kernel, so that it can be queued in the Xenomai
scheduler runqueues. That control block is semantically equivalent to
the "struct task_struct" type in the Linux kernel, but for representing
a thread that can be scheduled by the Xenomai kernel.

Threads created by any of the Xenomai APIs bind themselves to the
co-kernel when emerging (see the "trampoline" routines in the library
code), getting that special task control block addition in the process.
Once done, those threads can sleep on Xenomai resources. Regular (glibc)
threads don't bind themselves to Xenomai, so they may not sleep on such
resources, and will get EPERM if attempting to do so. That particular
action of extending a regular Linux thread with a Xenomai TCB is called
"shadowing" in our parlance, i.e. a Xenomai-specific shadow control
block gets added to the common Linux-originated task_struct.

However, many Xenomai services which only signal resources may be called
from a plain Linux context/thread, since the corresponding syscall does
not have to refer to the Xenomai control block of the caller for
carrying out the action. Typically, sem_post() is one of these.

>>
>>>
>>> I can do any conditional API usage through thread parameters to avoid calls which will not work for a relaxed thread, but I would like to retain the API (I guess I could switch to the posix skin but I would like to avoid another learning curve/test cycle for now)
>>>
>>> going forward/Xenomai 3: what would be your recommendation to address the issue long-term? 
>>>
>>
>> Xenomai 3 in dual kernel mode extends the feature above, with the
>> SCHED_WEAK class:
>> http://xenomai.org/migrating-from-xenomai-2-x-to-3-x/#Scheduling
>>
>>>
>>>
>>>
>>> (may sound like whacky question but great use case around here - turns out many jobs do not need RT capabilities but it would be handy to retain the flow)
>>>
>>
>> If I interpreted your question properly, it's definitely a legitimate
>> use case; sometimes the non-rt stuff may want to interact with the rt
>> world using Xenomai APIs, but without getting in the way priority-wise.
> 
> exactly. The unknown for me is: can that quasi-non-rt thread do more than a bona-fide RT thread. 

All Xenomai threads may call regular Linux services safely, the
co-kernel detects the situation and takes care of this. One just needs
to care about the symbol wrapping with the POSIX API, so that e.g.
connect() calls the Xenomai service for connecting a socket endpoint,
and __real_connect() calls the glibc version of that routine.

The point is about picking the most appropriate type of thread depending
on the expected duty:

- don't care for low latency, only regular Linux services needed =>
plain glibc/Linux threads
- don't care for low latency, may want to synchronize (i.e. block/sleep)
on Xenomai resources => SCHED_OTHER as implemented by the Xenomai
scheduler with 2.x
- do care for latency => any Xenomai policy but SCHED_OTHER (or
SCHED_WEAK in 3.x).

> 
> For comparison: in the RT-PREEMPT flavor of our stack, to achieve the same effect we'd just skip the sched_setscheduler(0, SCHED_FIFO, ..) step on thread creation and have the desired result.
>

That would be exactly the same with any Xenomai version and the POSIX
API. With the native(v2.x) or alchemy(v3.x) API, you would simply pass a
zero priority to rt_task_create/rt_task_shadow to leave the new thread
in the SCHED_OTHER class, instead of SCHED_FIFO.

> thanks for clarifying - that really helps!
> 

You are welcome.

-- 
Philippe.

Re: [Xenomai] mutate/start RT_TASK a posix thread?

[Michael Haberler]

Philippe,

this is excellent news because this means the scheme would work _and_ improve versatility.

I have only an optimization question left:

> Am 31.07.2015 um 10:22 schrieb Philippe Gerum <rpm@xenomai.org>:
> 
> On 07/31/2015 08:50 AM, Michael Haberler wrote:
>> Philippe,
>> 
>> 
>>> Am 30.07.2015 um 18:25 schrieb Philippe Gerum <rpm@xenomai.org>:
>>> 
>>> On 07/30/2015 07:26 AM, Michael Haberler wrote:
>>>> we're happily using RT threads using the Xenomai 2 thread API
>>>> 
>>>> is it possible _using this API_ to create/mutate/relax such a thread intentionally into a Posix thread but retaining the API usage?
>>> 
>>> It is possible to run them as low priority Xenomai threads, assigning
>>> them to the SCHED_OTHER class. The native API is retained for those
>>> threads, but their main runtime mode is relaxed, i.e. the co-kernel
>>> makes sure to switch them back to relaxed mode automatically before
>>> returning to user-space from a syscall which required a switch to
>>> primary mode.
>> 
>> that would give us 'low-priority RT (SCHED_OTHER)' threads which is already a great improvement for us, and it would be simple to do as you outlined in the followup mail
>> 
>> 
>> taking things one step further (I'm unsure about the precise semantics of 'relaxed mode', so far I understood it as 'behaves like a normal Linux thread'):
> 
> By relaxed, I mean scheduled by the regular kernel as any other
> non-Xenomai thread, not by the Xenomai scheduler. As a consequence of
> this, no rt guarantee, no pressure for very short response time.
> 
>> 
>> would such a SCHED_OTHER-class thread be able to use _any_ system calls like a normal Linux thread? like file I/O, sockets, poll, etc?
>> 
> 
> Yes. The only issue to care about is not passing file descriptors
> obtained by the Xenomai open() call to regular Linux I/O services. The
> converse works with the POSIX API though, since the Xenomai I/O
> subsystem hands over requests for fds it does not own to the
> corresponding glibc call.

not an issue since Xenomai file descriptors not used, but good to know

> 
>> if so - are there any side effects I should be aware of?
>> 
> 
> None. However, switching a non-rt Xenomai thread back and forth between
> the Xenomai and Linux sides involves several scheduling operations each
> time (one less for 3.x compared to 2.x though). Therefore, it is better
> if the low priority thread does not have to do this at a high rate, it's
> faily costly CPU-wise.

I just went through the code to see which native services are used post creation:

every thread cycle:
rt_task_wait_period() 
rt_timer_read().
rt_task_self() (unsure, need to check usage)

rarely:
rt_task_inquire()
rt_task_suspend()
rt_task_resume()

at thread exit: rt_task_delete() rt_task_join() but I guess not relevant

Assuming your load concern applies for these calls likewise, I see two options:

a) since the relax would happen intentionally through a startup option, I could have the thread check its descriptor and figure 'I better avoid native services' and use equivalent non-RT Linux services
b) if there is a cheap way of introspecting on the 'am I a relaxed thread?' property of a thread I could make the conditional API usage self-contained, i.e. without reference to our task descriptor; and it would help reducing API breakage.

I assume rt_task_self() is the right one. Do you think that is cheap enough to pursue option b) ?


this might be already rearranging deck chairs though - the question of CPU cost might not be that important after all:

the feature is intended to be used for throwing spare cores at jobs which are not time-critical, and they might use more of the Linux API while at it; but without having to rewrite too much

> 
>> (I guess my question really is: does 'relaxed mode equals Linux thread semantics' hold?)
>> 
>> 
> 
> Almost, with the added bonus of being able to synchronize with other
> Xenomai threads (rt and non-rt) as well, which a regular Linux thread can't.
> 
> The gist of the matter is about being able to block/sleep on Xenomai
> synchronization objects, such as sem4s, mutexes, queues, events, etc: to
> do that, the caller has to provide a Xenomai-originated task control
> block to the co-kernel, so that it can be queued in the Xenomai
> scheduler runqueues. That control block is semantically equivalent to
> the "struct task_struct" type in the Linux kernel, but for representing
> a thread that can be scheduled by the Xenomai kernel.
> 
> Threads created by any of the Xenomai APIs bind themselves to the
> co-kernel when emerging (see the "trampoline" routines in the library
> code), getting that special task control block addition in the process.
> Once done, those threads can sleep on Xenomai resources. Regular (glibc)
> threads don't bind themselves to Xenomai, so they may not sleep on such
> resources, and will get EPERM if attempting to do so. That particular
> action of extending a regular Linux thread with a Xenomai TCB is called
> "shadowing" in our parlance, i.e. a Xenomai-specific shadow control
> block gets added to the common Linux-originated task_struct.
> 
> However, many Xenomai services which only signal resources may be called
> from a plain Linux context/thread, since the corresponding syscall does
> not have to refer to the Xenomai control block of the caller for
> carrying out the action. Typically, sem_post() is one of these.
> 
>>> 
>>>> 
>>>> I can do any conditional API usage through thread parameters to avoid calls which will not work for a relaxed thread, but I would like to retain the API (I guess I could switch to the posix skin but I would like to avoid another learning curve/test cycle for now)
>>>> 
>>>> going forward/Xenomai 3: what would be your recommendation to address the issue long-term? 
>>>> 
>>> 
>>> Xenomai 3 in dual kernel mode extends the feature above, with the
>>> SCHED_WEAK class:
>>> http://xenomai.org/migrating-from-xenomai-2-x-to-3-x/#Scheduling
>>> 
>>>> 
>>>> 
>>>> 
>>>> (may sound like whacky question but great use case around here - turns out many jobs do not need RT capabilities but it would be handy to retain the flow)
>>>> 
>>> 
>>> If I interpreted your question properly, it's definitely a legitimate
>>> use case; sometimes the non-rt stuff may want to interact with the rt
>>> world using Xenomai APIs, but without getting in the way priority-wise.
>> 
>> exactly. The unknown for me is: can that quasi-non-rt thread do more than a bona-fide RT thread. 
> 
> All Xenomai threads may call regular Linux services safely, the
> co-kernel detects the situation and takes care of this. One just needs
> to care about the symbol wrapping with the POSIX API, so that e.g.
> connect() calls the Xenomai service for connecting a socket endpoint,
> and __real_connect() calls the glibc version of that routine.
> 
> The point is about picking the most appropriate type of thread depending
> on the expected duty:
> 
> - don't care for low latency, only regular Linux services needed =>
> plain glibc/Linux threads
> - don't care for low latency, may want to synchronize (i.e. block/sleep)
> on Xenomai resources => SCHED_OTHER as implemented by the Xenomai
> scheduler with 2.x
> - do care for latency => any Xenomai policy but SCHED_OTHER (or
> SCHED_WEAK in 3.x).

well in this case it's the first option plus 'programmer too lazy to rewrite' ;)

thanks - that really helped my understanding how this all works!

I'll report what comes out of this

best regards

Micahel


> 
>> 
>> For comparison: in the RT-PREEMPT flavor of our stack, to achieve the same effect we'd just skip the sched_setscheduler(0, SCHED_FIFO, ..) step on thread creation and have the desired result.
>> 
> 
> That would be exactly the same with any Xenomai version and the POSIX
> API. With the native(v2.x) or alchemy(v3.x) API, you would simply pass a
> zero priority to rt_task_create/rt_task_shadow to leave the new thread
> in the SCHED_OTHER class, instead of SCHED_FIFO.
> 
>> thanks for clarifying - that really helps!
>> 
> 
> You are welcome.
> 
> -- 
> Philippe.

Re: [Xenomai] mutate/start RT_TASK a posix thread?

[Philippe Gerum]

On 07/31/2015 11:21 AM, Michael Haberler wrote:
> Philippe,
> 
> this is excellent news because this means the scheme would work _and_ improve versatility.
> 
> I have only an optimization question left:
> 
>> Am 31.07.2015 um 10:22 schrieb Philippe Gerum <rpm@xenomai.org>:
>>
>> On 07/31/2015 08:50 AM, Michael Haberler wrote:
>>> Philippe,
>>>
>>>
>>>> Am 30.07.2015 um 18:25 schrieb Philippe Gerum <rpm@xenomai.org>:
>>>>
>>>> On 07/30/2015 07:26 AM, Michael Haberler wrote:
>>>>> we're happily using RT threads using the Xenomai 2 thread API
>>>>>
>>>>> is it possible _using this API_ to create/mutate/relax such a thread intentionally into a Posix thread but retaining the API usage?
>>>>
>>>> It is possible to run them as low priority Xenomai threads, assigning
>>>> them to the SCHED_OTHER class. The native API is retained for those
>>>> threads, but their main runtime mode is relaxed, i.e. the co-kernel
>>>> makes sure to switch them back to relaxed mode automatically before
>>>> returning to user-space from a syscall which required a switch to
>>>> primary mode.
>>>
>>> that would give us 'low-priority RT (SCHED_OTHER)' threads which is already a great improvement for us, and it would be simple to do as you outlined in the followup mail
>>>
>>>
>>> taking things one step further (I'm unsure about the precise semantics of 'relaxed mode', so far I understood it as 'behaves like a normal Linux thread'):
>>
>> By relaxed, I mean scheduled by the regular kernel as any other
>> non-Xenomai thread, not by the Xenomai scheduler. As a consequence of
>> this, no rt guarantee, no pressure for very short response time.
>>
>>>
>>> would such a SCHED_OTHER-class thread be able to use _any_ system calls like a normal Linux thread? like file I/O, sockets, poll, etc?
>>>
>>
>> Yes. The only issue to care about is not passing file descriptors
>> obtained by the Xenomai open() call to regular Linux I/O services. The
>> converse works with the POSIX API though, since the Xenomai I/O
>> subsystem hands over requests for fds it does not own to the
>> corresponding glibc call.
> 
> not an issue since Xenomai file descriptors not used, but good to know
> 
>>
>>> if so - are there any side effects I should be aware of?
>>>
>>
>> None. However, switching a non-rt Xenomai thread back and forth between
>> the Xenomai and Linux sides involves several scheduling operations each
>> time (one less for 3.x compared to 2.x though). Therefore, it is better
>> if the low priority thread does not have to do this at a high rate, it's
>> faily costly CPU-wise.
> 
> I just went through the code to see which native services are used post creation:
> 
> every thread cycle:
> rt_task_wait_period() 
> rt_timer_read().
> rt_task_self() (unsure, need to check usage)
> 
> rarely:
> rt_task_inquire()
> rt_task_suspend()
> rt_task_resume()
> 
> at thread exit: rt_task_delete() rt_task_join() but I guess not relevant
> 
> Assuming your load concern applies for these calls likewise, I see two options:
> 
> a) since the relax would happen intentionally through a startup option, I could have the thread check its descriptor and figure 'I better avoid native services' and use equivalent non-RT Linux services
> b) if there is a cheap way of introspecting on the 'am I a relaxed thread?' property of a thread I could make the conditional API usage self-contained, i.e. without reference to our task descriptor; and it would help reducing API breakage.
> 
> I assume rt_task_self() is the right one. Do you think that is cheap enough to pursue option b) ?
> 
> 

rt_task_self() would return a valid descriptor for both SCHED_OTHER and
SCHED_FIFO threads, provided they are Xenomai threads. You need
something cheap that would quickly tell the caller about its
capabilities as a Xenomai thread instead.

With 2.x, you could hack that check with xeno_get_current_mode(), which
is part of the internal API, testing the XNOTHER bit there. Have a look
at src/skins/native/mutex.c for some examples. XNOTHER is armed for
Xenomai-managed SCHED_OTHER threads. This only entails a plain memory
reference, so this is cheap enough. Granted, this is an internal call,
so no guarantee for backward compat. But Xenomai 2.x is virtually EOL
already, there will be no 2.7, so that feature is there to stay
indefinitely.

-- 
Philippe.

Re: [Xenomai] mutate/start RT_TASK a posix thread?

[Michael Haberler]

> Am 31.07.2015 um 16:07 schrieb Philippe Gerum <rpm@xenomai.org>:
> 
> On 07/31/2015 11:21 AM, Michael Haberler wrote:
>> Philippe,
>> 
>> this is excellent news because this means the scheme would work _and_ improve versatility.
>> 
>> I have only an optimization question left:
>> 
>>> Am 31.07.2015 um 10:22 schrieb Philippe Gerum <rpm@xenomai.org>:
>>> 
>>> On 07/31/2015 08:50 AM, Michael Haberler wrote:
>>>> Philippe,
>>>> 
>>>> 
>>>>> Am 30.07.2015 um 18:25 schrieb Philippe Gerum <rpm@xenomai.org>:
>>>>> 
>>>>> On 07/30/2015 07:26 AM, Michael Haberler wrote:
>>>>>> we're happily using RT threads using the Xenomai 2 thread API
>>>>>> 
>>>>>> is it possible _using this API_ to create/mutate/relax such a thread intentionally into a Posix thread but retaining the API usage?
>>>>> 
>>>>> It is possible to run them as low priority Xenomai threads, assigning
>>>>> them to the SCHED_OTHER class. The native API is retained for those
>>>>> threads, but their main runtime mode is relaxed, i.e. the co-kernel
>>>>> makes sure to switch them back to relaxed mode automatically before
>>>>> returning to user-space from a syscall which required a switch to
>>>>> primary mode.
>>>> 
>>>> that would give us 'low-priority RT (SCHED_OTHER)' threads which is already a great improvement for us, and it would be simple to do as you outlined in the followup mail
>>>> 
>>>> 
>>>> taking things one step further (I'm unsure about the precise semantics of 'relaxed mode', so far I understood it as 'behaves like a normal Linux thread'):
>>> 
>>> By relaxed, I mean scheduled by the regular kernel as any other
>>> non-Xenomai thread, not by the Xenomai scheduler. As a consequence of
>>> this, no rt guarantee, no pressure for very short response time.
>>> 
>>>> 
>>>> would such a SCHED_OTHER-class thread be able to use _any_ system calls like a normal Linux thread? like file I/O, sockets, poll, etc?
>>>> 
>>> 
>>> Yes. The only issue to care about is not passing file descriptors
>>> obtained by the Xenomai open() call to regular Linux I/O services. The
>>> converse works with the POSIX API though, since the Xenomai I/O
>>> subsystem hands over requests for fds it does not own to the
>>> corresponding glibc call.
>> 
>> not an issue since Xenomai file descriptors not used, but good to know
>> 
>>> 
>>>> if so - are there any side effects I should be aware of?
>>>> 
>>> 
>>> None. However, switching a non-rt Xenomai thread back and forth between
>>> the Xenomai and Linux sides involves several scheduling operations each
>>> time (one less for 3.x compared to 2.x though). Therefore, it is better
>>> if the low priority thread does not have to do this at a high rate, it's
>>> faily costly CPU-wise.
>> 
>> I just went through the code to see which native services are used post creation:
>> 
>> every thread cycle:
>> rt_task_wait_period() 
>> rt_timer_read().
>> rt_task_self() (unsure, need to check usage)
>> 
>> rarely:
>> rt_task_inquire()
>> rt_task_suspend()
>> rt_task_resume()
>> 
>> at thread exit: rt_task_delete() rt_task_join() but I guess not relevant
>> 
>> Assuming your load concern applies for these calls likewise, I see two options:
>> 
>> a) since the relax would happen intentionally through a startup option, I could have the thread check its descriptor and figure 'I better avoid native services' and use equivalent non-RT Linux services
>> b) if there is a cheap way of introspecting on the 'am I a relaxed thread?' property of a thread I could make the conditional API usage self-contained, i.e. without reference to our task descriptor; and it would help reducing API breakage.
>> 
>> I assume rt_task_self() is the right one. Do you think that is cheap enough to pursue option b) ?
>> 
>> 
> 
> rt_task_self() would return a valid descriptor for both SCHED_OTHER and
> SCHED_FIFO threads, provided they are Xenomai threads. You need
> something cheap that would quickly tell the caller about its
> capabilities as a Xenomai thread instead.
> 
> With 2.x, you could hack that check with xeno_get_current_mode(), which
> is part of the internal API, testing the XNOTHER bit there. Have a look
> at src/skins/native/mutex.c for some examples. XNOTHER is armed for
> Xenomai-managed SCHED_OTHER threads. This only entails a plain memory
> reference, so this is cheap enough. Granted, this is an internal call,
> so no guarantee for backward compat. But Xenomai 2.x is virtually EOL
> already, there will be no 2.7, so that feature is there to stay
> indefinitely.

fine, I'll manage that.

any suggestions for Xenomai 3? If too hairy, I'll go for the 'inspect our own thread descriptor' option.

thanks,

- Michael

> 
> -- 
> Philippe.

Re: [Xenomai] mutate/start RT_TASK a posix thread?

[Philippe Gerum]

On 07/31/2015 05:17 PM, Michael Haberler wrote:
> 
>> Am 31.07.2015 um 16:07 schrieb Philippe Gerum <rpm@xenomai.org>:
>>
>> On 07/31/2015 11:21 AM, Michael Haberler wrote:
>>> Philippe,
>>>
>>> this is excellent news because this means the scheme would work _and_ improve versatility.
>>>
>>> I have only an optimization question left:
>>>
>>>> Am 31.07.2015 um 10:22 schrieb Philippe Gerum <rpm@xenomai.org>:
>>>>
>>>> On 07/31/2015 08:50 AM, Michael Haberler wrote:
>>>>> Philippe,
>>>>>
>>>>>
>>>>>> Am 30.07.2015 um 18:25 schrieb Philippe Gerum <rpm@xenomai.org>:
>>>>>>
>>>>>> On 07/30/2015 07:26 AM, Michael Haberler wrote:
>>>>>>> we're happily using RT threads using the Xenomai 2 thread API
>>>>>>>
>>>>>>> is it possible _using this API_ to create/mutate/relax such a thread intentionally into a Posix thread but retaining the API usage?
>>>>>>
>>>>>> It is possible to run them as low priority Xenomai threads, assigning
>>>>>> them to the SCHED_OTHER class. The native API is retained for those
>>>>>> threads, but their main runtime mode is relaxed, i.e. the co-kernel
>>>>>> makes sure to switch them back to relaxed mode automatically before
>>>>>> returning to user-space from a syscall which required a switch to
>>>>>> primary mode.
>>>>>
>>>>> that would give us 'low-priority RT (SCHED_OTHER)' threads which is already a great improvement for us, and it would be simple to do as you outlined in the followup mail
>>>>>
>>>>>
>>>>> taking things one step further (I'm unsure about the precise semantics of 'relaxed mode', so far I understood it as 'behaves like a normal Linux thread'):
>>>>
>>>> By relaxed, I mean scheduled by the regular kernel as any other
>>>> non-Xenomai thread, not by the Xenomai scheduler. As a consequence of
>>>> this, no rt guarantee, no pressure for very short response time.
>>>>
>>>>>
>>>>> would such a SCHED_OTHER-class thread be able to use _any_ system calls like a normal Linux thread? like file I/O, sockets, poll, etc?
>>>>>
>>>>
>>>> Yes. The only issue to care about is not passing file descriptors
>>>> obtained by the Xenomai open() call to regular Linux I/O services. The
>>>> converse works with the POSIX API though, since the Xenomai I/O
>>>> subsystem hands over requests for fds it does not own to the
>>>> corresponding glibc call.
>>>
>>> not an issue since Xenomai file descriptors not used, but good to know
>>>
>>>>
>>>>> if so - are there any side effects I should be aware of?
>>>>>
>>>>
>>>> None. However, switching a non-rt Xenomai thread back and forth between
>>>> the Xenomai and Linux sides involves several scheduling operations each
>>>> time (one less for 3.x compared to 2.x though). Therefore, it is better
>>>> if the low priority thread does not have to do this at a high rate, it's
>>>> faily costly CPU-wise.
>>>
>>> I just went through the code to see which native services are used post creation:
>>>
>>> every thread cycle:
>>> rt_task_wait_period() 
>>> rt_timer_read().
>>> rt_task_self() (unsure, need to check usage)
>>>
>>> rarely:
>>> rt_task_inquire()
>>> rt_task_suspend()
>>> rt_task_resume()
>>>
>>> at thread exit: rt_task_delete() rt_task_join() but I guess not relevant
>>>
>>> Assuming your load concern applies for these calls likewise, I see two options:
>>>
>>> a) since the relax would happen intentionally through a startup option, I could have the thread check its descriptor and figure 'I better avoid native services' and use equivalent non-RT Linux services
>>> b) if there is a cheap way of introspecting on the 'am I a relaxed thread?' property of a thread I could make the conditional API usage self-contained, i.e. without reference to our task descriptor; and it would help reducing API breakage.
>>>
>>> I assume rt_task_self() is the right one. Do you think that is cheap enough to pursue option b) ?
>>>
>>>
>>
>> rt_task_self() would return a valid descriptor for both SCHED_OTHER and
>> SCHED_FIFO threads, provided they are Xenomai threads. You need
>> something cheap that would quickly tell the caller about its
>> capabilities as a Xenomai thread instead.
>>
>> With 2.x, you could hack that check with xeno_get_current_mode(), which
>> is part of the internal API, testing the XNOTHER bit there. Have a look
>> at src/skins/native/mutex.c for some examples. XNOTHER is armed for
>> Xenomai-managed SCHED_OTHER threads. This only entails a plain memory
>> reference, so this is cheap enough. Granted, this is an internal call,
>> so no guarantee for backward compat. But Xenomai 2.x is virtually EOL
>> already, there will be no 2.7, so that feature is there to stay
>> indefinitely.
> 
> fine, I'll manage that.
> 
> any suggestions for Xenomai 3? If too hairy, I'll go for the 'inspect our own thread descriptor' option.
> 

3.x has cobalt_get_current_mode() from lib/cobalt/current.h, testing for
XNWEAK will do the trick.

-- 
Philippe.

Re: [Xenomai] mutate/start RT_TASK a posix thread?

[Philippe Gerum]

On 07/31/2015 05:35 PM, Philippe Gerum wrote:
> On 07/31/2015 05:17 PM, Michael Haberler wrote:
>>
>>> Am 31.07.2015 um 16:07 schrieb Philippe Gerum <rpm@xenomai.org>:
>>>
>>> On 07/31/2015 11:21 AM, Michael Haberler wrote:
>>>> Philippe,
>>>>
>>>> this is excellent news because this means the scheme would work _and_ improve versatility.
>>>>
>>>> I have only an optimization question left:
>>>>
>>>>> Am 31.07.2015 um 10:22 schrieb Philippe Gerum <rpm@xenomai.org>:
>>>>>
>>>>> On 07/31/2015 08:50 AM, Michael Haberler wrote:
>>>>>> Philippe,
>>>>>>
>>>>>>
>>>>>>> Am 30.07.2015 um 18:25 schrieb Philippe Gerum <rpm@xenomai.org>:
>>>>>>>
>>>>>>> On 07/30/2015 07:26 AM, Michael Haberler wrote:
>>>>>>>> we're happily using RT threads using the Xenomai 2 thread API
>>>>>>>>
>>>>>>>> is it possible _using this API_ to create/mutate/relax such a thread intentionally into a Posix thread but retaining the API usage?
>>>>>>>
>>>>>>> It is possible to run them as low priority Xenomai threads, assigning
>>>>>>> them to the SCHED_OTHER class. The native API is retained for those
>>>>>>> threads, but their main runtime mode is relaxed, i.e. the co-kernel
>>>>>>> makes sure to switch them back to relaxed mode automatically before
>>>>>>> returning to user-space from a syscall which required a switch to
>>>>>>> primary mode.
>>>>>>
>>>>>> that would give us 'low-priority RT (SCHED_OTHER)' threads which is already a great improvement for us, and it would be simple to do as you outlined in the followup mail
>>>>>>
>>>>>>
>>>>>> taking things one step further (I'm unsure about the precise semantics of 'relaxed mode', so far I understood it as 'behaves like a normal Linux thread'):
>>>>>
>>>>> By relaxed, I mean scheduled by the regular kernel as any other
>>>>> non-Xenomai thread, not by the Xenomai scheduler. As a consequence of
>>>>> this, no rt guarantee, no pressure for very short response time.
>>>>>
>>>>>>
>>>>>> would such a SCHED_OTHER-class thread be able to use _any_ system calls like a normal Linux thread? like file I/O, sockets, poll, etc?
>>>>>>
>>>>>
>>>>> Yes. The only issue to care about is not passing file descriptors
>>>>> obtained by the Xenomai open() call to regular Linux I/O services. The
>>>>> converse works with the POSIX API though, since the Xenomai I/O
>>>>> subsystem hands over requests for fds it does not own to the
>>>>> corresponding glibc call.
>>>>
>>>> not an issue since Xenomai file descriptors not used, but good to know
>>>>
>>>>>
>>>>>> if so - are there any side effects I should be aware of?
>>>>>>
>>>>>
>>>>> None. However, switching a non-rt Xenomai thread back and forth between
>>>>> the Xenomai and Linux sides involves several scheduling operations each
>>>>> time (one less for 3.x compared to 2.x though). Therefore, it is better
>>>>> if the low priority thread does not have to do this at a high rate, it's
>>>>> faily costly CPU-wise.
>>>>
>>>> I just went through the code to see which native services are used post creation:
>>>>
>>>> every thread cycle:
>>>> rt_task_wait_period() 
>>>> rt_timer_read().
>>>> rt_task_self() (unsure, need to check usage)
>>>>
>>>> rarely:
>>>> rt_task_inquire()
>>>> rt_task_suspend()
>>>> rt_task_resume()
>>>>
>>>> at thread exit: rt_task_delete() rt_task_join() but I guess not relevant
>>>>
>>>> Assuming your load concern applies for these calls likewise, I see two options:
>>>>
>>>> a) since the relax would happen intentionally through a startup option, I could have the thread check its descriptor and figure 'I better avoid native services' and use equivalent non-RT Linux services
>>>> b) if there is a cheap way of introspecting on the 'am I a relaxed thread?' property of a thread I could make the conditional API usage self-contained, i.e. without reference to our task descriptor; and it would help reducing API breakage.
>>>>
>>>> I assume rt_task_self() is the right one. Do you think that is cheap enough to pursue option b) ?
>>>>
>>>>
>>>
>>> rt_task_self() would return a valid descriptor for both SCHED_OTHER and
>>> SCHED_FIFO threads, provided they are Xenomai threads. You need
>>> something cheap that would quickly tell the caller about its
>>> capabilities as a Xenomai thread instead.
>>>
>>> With 2.x, you could hack that check with xeno_get_current_mode(), which
>>> is part of the internal API, testing the XNOTHER bit there. Have a look
>>> at src/skins/native/mutex.c for some examples. XNOTHER is armed for
>>> Xenomai-managed SCHED_OTHER threads. This only entails a plain memory
>>> reference, so this is cheap enough. Granted, this is an internal call,
>>> so no guarantee for backward compat. But Xenomai 2.x is virtually EOL
>>> already, there will be no 2.7, so that feature is there to stay
>>> indefinitely.
>>
>> fine, I'll manage that.
>>
>> any suggestions for Xenomai 3? If too hairy, I'll go for the 'inspect our own thread descriptor' option.
>>
> 
> 3.x has cobalt_get_current_mode() from lib/cobalt/current.h, testing for
> XNWEAK will do the trick.
> 

I have just exported cobalt_thread_mode() in the upcoming 3.0, which
returns the runtime mode/state bits for the caller. This is still part
of an internal API, so there won't be any ABI/API compat guarantees with
future releases. This said, I don't see how libcobalt could do without
it either ATM.

Unlike lib/cobalt/current.h which is not installed, sys/cobalt.h is and
including it will pull the proper declarations for any application to
use cobalt_thread_mode().

-- 
Philippe.

Re: [Xenomai] mutate/start RT_TASK a posix thread?

[Michael Haberler]

> Am 31.07.2015 um 17:53 schrieb Philippe Gerum <rpm@xenomai.org>:
> 
> On 07/31/2015 05:35 PM, Philippe Gerum wrote:
>> On 07/31/2015 05:17 PM, Michael Haberler wrote:
>>> 
>>>> Am 31.07.2015 um 16:07 schrieb Philippe Gerum <rpm@xenomai.org>:
>>>> 
>>>> On 07/31/2015 11:21 AM, Michael Haberler wrote:
>>>>> Philippe,
>>>>> 
>>>>> this is excellent news because this means the scheme would work _and_ improve versatility.
>>>>> 
>>>>> I have only an optimization question left:
>>>>> 
>>>>>> Am 31.07.2015 um 10:22 schrieb Philippe Gerum <rpm@xenomai.org>:
>>>>>> 
>>>>>> On 07/31/2015 08:50 AM, Michael Haberler wrote:
>>>>>>> Philippe,
>>>>>>> 
>>>>>>> 
>>>>>>>> Am 30.07.2015 um 18:25 schrieb Philippe Gerum <rpm@xenomai.org>:
>>>>>>>> 
>>>>>>>> On 07/30/2015 07:26 AM, Michael Haberler wrote:
>>>>>>>>> we're happily using RT threads using the Xenomai 2 thread API
>>>>>>>>> 
>>>>>>>>> is it possible _using this API_ to create/mutate/relax such a thread intentionally into a Posix thread but retaining the API usage?
>>>>>>>> 
>>>>>>>> It is possible to run them as low priority Xenomai threads, assigning
>>>>>>>> them to the SCHED_OTHER class. The native API is retained for those
>>>>>>>> threads, but their main runtime mode is relaxed, i.e. the co-kernel
>>>>>>>> makes sure to switch them back to relaxed mode automatically before
>>>>>>>> returning to user-space from a syscall which required a switch to
>>>>>>>> primary mode.
>>>>>>> 
>>>>>>> that would give us 'low-priority RT (SCHED_OTHER)' threads which is already a great improvement for us, and it would be simple to do as you outlined in the followup mail
>>>>>>> 
>>>>>>> 
>>>>>>> taking things one step further (I'm unsure about the precise semantics of 'relaxed mode', so far I understood it as 'behaves like a normal Linux thread'):
>>>>>> 
>>>>>> By relaxed, I mean scheduled by the regular kernel as any other
>>>>>> non-Xenomai thread, not by the Xenomai scheduler. As a consequence of
>>>>>> this, no rt guarantee, no pressure for very short response time.
>>>>>> 
>>>>>>> 
>>>>>>> would such a SCHED_OTHER-class thread be able to use _any_ system calls like a normal Linux thread? like file I/O, sockets, poll, etc?
>>>>>>> 
>>>>>> 
>>>>>> Yes. The only issue to care about is not passing file descriptors
>>>>>> obtained by the Xenomai open() call to regular Linux I/O services. The
>>>>>> converse works with the POSIX API though, since the Xenomai I/O
>>>>>> subsystem hands over requests for fds it does not own to the
>>>>>> corresponding glibc call.
>>>>> 
>>>>> not an issue since Xenomai file descriptors not used, but good to know
>>>>> 
>>>>>> 
>>>>>>> if so - are there any side effects I should be aware of?
>>>>>>> 
>>>>>> 
>>>>>> None. However, switching a non-rt Xenomai thread back and forth between
>>>>>> the Xenomai and Linux sides involves several scheduling operations each
>>>>>> time (one less for 3.x compared to 2.x though). Therefore, it is better
>>>>>> if the low priority thread does not have to do this at a high rate, it's
>>>>>> faily costly CPU-wise.
>>>>> 
>>>>> I just went through the code to see which native services are used post creation:
>>>>> 
>>>>> every thread cycle:
>>>>> rt_task_wait_period() 
>>>>> rt_timer_read().
>>>>> rt_task_self() (unsure, need to check usage)
>>>>> 
>>>>> rarely:
>>>>> rt_task_inquire()
>>>>> rt_task_suspend()
>>>>> rt_task_resume()
>>>>> 
>>>>> at thread exit: rt_task_delete() rt_task_join() but I guess not relevant
>>>>> 
>>>>> Assuming your load concern applies for these calls likewise, I see two options:
>>>>> 
>>>>> a) since the relax would happen intentionally through a startup option, I could have the thread check its descriptor and figure 'I better avoid native services' and use equivalent non-RT Linux services
>>>>> b) if there is a cheap way of introspecting on the 'am I a relaxed thread?' property of a thread I could make the conditional API usage self-contained, i.e. without reference to our task descriptor; and it would help reducing API breakage.
>>>>> 
>>>>> I assume rt_task_self() is the right one. Do you think that is cheap enough to pursue option b) ?
>>>>> 
>>>>> 
>>>> 
>>>> rt_task_self() would return a valid descriptor for both SCHED_OTHER and
>>>> SCHED_FIFO threads, provided they are Xenomai threads. You need
>>>> something cheap that would quickly tell the caller about its
>>>> capabilities as a Xenomai thread instead.
>>>> 
>>>> With 2.x, you could hack that check with xeno_get_current_mode(), which
>>>> is part of the internal API, testing the XNOTHER bit there. Have a look
>>>> at src/skins/native/mutex.c for some examples. XNOTHER is armed for
>>>> Xenomai-managed SCHED_OTHER threads. This only entails a plain memory
>>>> reference, so this is cheap enough. Granted, this is an internal call,
>>>> so no guarantee for backward compat. But Xenomai 2.x is virtually EOL
>>>> already, there will be no 2.7, so that feature is there to stay
>>>> indefinitely.
>>> 
>>> fine, I'll manage that.
>>> 
>>> any suggestions for Xenomai 3? If too hairy, I'll go for the 'inspect our own thread descriptor' option.
>>> 
>> 
>> 3.x has cobalt_get_current_mode() from lib/cobalt/current.h, testing for
>> XNWEAK will do the trick.
>> 
> 
> I have just exported cobalt_thread_mode() in the upcoming 3.0, which
> returns the runtime mode/state bits for the caller. This is still part
> of an internal API, so there won't be any ABI/API compat guarantees with
> future releases. This said, I don't see how libcobalt could do without
> it either ATM.

excellent! a bit of shaky ground is just fine in such a major transition.

thanks!

- Michael

> 
> Unlike lib/cobalt/current.h which is not installed, sys/cobalt.h is and
> including it will pull the proper declarations for any application to
> use cobalt_thread_mode().
> 
> -- 
> Philippe.