[Xenomai-help] Xenomai and futexes - Native API optimized for user space only applications

[Xenomai-help] Xenomai and futexes - Native API optimized for user space only applications

[M. Koehrer]

Hi everybody,

I am using Xenomai for a high performance real time simulation system. 
All of the simulation code is executed within user space. One application is running that consists
of several Xenomai real time threads.
For performance reasons I am always using the latest PC technology available (which is currently
Pentium D or Core2Duo PCs, we are thinking even of using quad-core CPUs).
There has to be a kind of thread synchronisation e.g. when accessing shared data.
Hardware I/O is done via rtnet or via PCI I/O boards that work in user space aswell (PCI memory
mapped into the user space).
This is done e.g. by using semaphores, mutexes et.c
I like the Xenomai-native skin as it provides a very clear API that is easy to use.
However, for a user space only application it is a performance killer, that all API calls
lead to a mode switch from user to kernel space. Each API call takes about 1-2 microseconds (us)
on my PC which is really expensive.
Especially when inter process communication is used to protect the access to shared data
it is mostly the case that the calling thread does not have to wait. In this situation there is no
need for a context switch. The API call did not lead to a rescheduling of the available tasks.
And for this the required 1-2 us do really hurt.

Thus my question/proposal is if there is a plan to use a "variant" of the native API that is optimized for
user space only applications. In this case e.g. futexes can be used. If there is a need to
reschedule to another task it is fine to "invest" the 2us but it can be avoided mostly which should
increase the overall performance dramatically.
This would lead to a library where a big part of the functionality is handled directly in the library 
(in user space). Currently the skin passes the (user space) API call via a Xenomai System call 
to the kernel space to execute there the actual functionality.

Thanks for any feedback on this proposal.

Regards

Mathias





-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] Xenomai and futexes - Native API optimized for user space only applications

[Dmitry Adamushko]

On 09/05/07, M. Koehrer <mathias_koehrer@domain.hid> wrote:
> [...]
>
> Thus my question/proposal is if there is a plan to use a "variant" of the native API that is optimized for
> user space only applications. In this case e.g. futexes can be used. If there is a need to
> reschedule to another task it is fine to "invest" the 2us but it can be avoided mostly which should
> increase the overall performance dramatically.

If it's always granted user-space tasks enter these lock/unlock calls
_only_ from the primary domain, it should be easy to implement (+ a
variant of atomic compare_and_swap operation for a given arch is
available). e.g. look at how linuxthreads does it e.g. in uClibc.

Given an additional restriction above (also quite bearable for some
apps. I guess) it likely would lead to introducing a separate type of
lock (maybe a sub-type of the existing).. not sure, consider it only
as brief comments at the moment.


-- 
Best regards,
Dmitry Adamushko

Re: [Xenomai-help] Xenomai and futexes - Native API optimized for user space only applications

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2394 bytes --]

M. Koehrer wrote:
> Hi everybody,
> 
> I am using Xenomai for a high performance real time simulation system. 
> All of the simulation code is executed within user space. One application is running that consists
> of several Xenomai real time threads.
> For performance reasons I am always using the latest PC technology available (which is currently
> Pentium D or Core2Duo PCs, we are thinking even of using quad-core CPUs).
> There has to be a kind of thread synchronisation e.g. when accessing shared data.
> Hardware I/O is done via rtnet or via PCI I/O boards that work in user space aswell (PCI memory
> mapped into the user space).
> This is done e.g. by using semaphores, mutexes et.c
> I like the Xenomai-native skin as it provides a very clear API that is easy to use.
> However, for a user space only application it is a performance killer, that all API calls
> lead to a mode switch from user to kernel space. Each API call takes about 1-2 microseconds (us)
> on my PC which is really expensive.
> Especially when inter process communication is used to protect the access to shared data
> it is mostly the case that the calling thread does not have to wait. In this situation there is no
> need for a context switch. The API call did not lead to a rescheduling of the available tasks.
> And for this the required 1-2 us do really hurt.
> 
> Thus my question/proposal is if there is a plan to use a "variant" of the native API that is optimized for
> user space only applications. In this case e.g. futexes can be used. If there is a need to
> reschedule to another task it is fine to "invest" the 2us but it can be avoided mostly which should
> increase the overall performance dramatically.

Your analysis is not wrong. But before going into any detail may I ask
you for on clarification: What do you want to optimise *primarily*, the
average execution time of your RT threads in order to gain more CPU time
for non-RT Linux jobs *or* the worst case execution time of your RT threads?

> This would lead to a library where a big part of the functionality is handled directly in the library 
> (in user space). Currently the skin passes the (user space) API call via a Xenomai System call 
> to the kernel space to execute there the actual functionality.
> 
> Thanks for any feedback on this proposal.
> 
> Regards
> 
> Mathias
> 

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] Xenomai and futexes - Native API optimized for

[M. Koehrer]

Hi Jan,
 
primarily I want to execute my simulation models as fast as possible. This is, I want 
to optimize the average execution time. As there are a series of critical sections to
be passed this should also have an impact on the worst execution time.

The critical region in the simulation are at locations when a lower prio thread and a higher
prio thread access the same data. This happens at a couple of times within a simulation step 
(within that critical region there are typically some copy operations).
However, once the higher prio thread has entered one critical region it is sure (at least if
both threads run on the same CPU core) that it will be able to enter the next critical region without
waiting.
That means, that improving the average runtime should also help to improve the worst case
runtime of the highest prio thread.

Regards

Mathias
> 
> Your analysis is not wrong. But before going into any detail may I ask
> you for on clarification: What do you want to optimise *primarily*, the
> average execution time of your RT threads in order to gain more CPU time
> for non-RT Linux jobs *or* the worst case execution time of your RT
> threads?
> 


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] Xenomai and futexes - Native API optimized for

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1447 bytes --]

M. Koehrer wrote:
> Hi Jan,
>  
> primarily I want to execute my simulation models as fast as possible. This is, I want 
> to optimize the average execution time. As there are a series of critical sections to
> be passed this should also have an impact on the worst execution time.
> 
> The critical region in the simulation are at locations when a lower prio thread and a higher
> prio thread access the same data. This happens at a couple of times within a simulation step 
> (within that critical region there are typically some copy operations).
> However, once the higher prio thread has entered one critical region it is sure (at least if
> both threads run on the same CPU core) that it will be able to enter the next critical region without
> waiting.
> That means, that improving the average runtime should also help to improve the worst case
> runtime of the highest prio thread.

Not saying that the futex idea itself is not worth pursuing, but I would
first of all think about if there aren't other, far simpler
optimisations available at application level: lock-less algorithms e.g.
that flip some current-buffer pointers instead of locking the access to
a single shared buffer. Or a locking scheme that reduces the number of
locks the highest prio thread has to take in a row. Specifically when
you say that taking lock A implies to succeed in taking B as well, I
wonder if there is a need for A!=B.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] Xenomai and futexes - Native API optimized for

[Herman Bruyninckx]

On Wed, 9 May 2007, M. Koehrer wrote:

[...]
> The critical region in the simulation are at locations when a lower prio
> thread and a higher prio thread access the same data. This happens at a
> couple of times within a simulation step (within that critical region
> there are typically some copy operations).

I agree with Jan Kiszka (in another reply in this thread) that it's
probably better to improve the design of your application than to shift the
problem to asking for "yet another feature" of the RTOS!

In our robot control framework (orocos.org) that works as a user space
application on top of Xenomai we have implemented lock-free data exchange,
which does'nt require involvement of the scheduler at all, and which
improves the timing performance of the highest priority thread. It might be
useful for you to discuss this implementation aspect on the orocos
developers mailinglist...

Herman Bruyninckx

--
   K.U.Leuven, Mechanical Eng.,  Mechatronics & Robotics Research Group
      <http://people.mech.kuleuven.be/~bruyninc> Tel: +32 16 322480
   Coordinator EURON (European Robotics Research Network)
      <http://www.euron.org>

Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm

Re: [Xenomai-help] Xenomai and futexes - Native API optimized for

[M. Koehrer]

Hi Herman and Jan,

thanks for your replies.
I agree that is the best to do an approach that avoids critical sections.
However, we have here the typical "historical heritage". Parts of this code have
been ported from another system (where they worked with cli/sti).
Other parts are generated from a third party code generator.
This code generator (which is not in my hand) is designed primarily
for offline simulation within a single task.
Thus, the guys that programmed that, did not care about multi thread programming.
The programs write the results into output variables.
To avoid critical section I have now to do either
an additional copy (also quite expensive) of the output data to a location 
where I can work without any risk of being interrupted or I lock the access
 to the data.
That means either I do lots of additional copy actions (also quite expensive) or I work
with the critical region.

Thus for me it is not an either - or - option.
I can not optimize the code to be 100% free of critical sections (however it can be done in
some parts..). That means I still have to count on the IPC provided by the OS.

Regards

Mathias

> > The critical region in the simulation are at locations when a lower prio
> > thread and a higher prio thread access the same data. This happens at a
> > couple of times within a simulation step (within that critical region
> > there are typically some copy operations).
> 
> I agree with Jan Kiszka (in another reply in this thread) that it's
> probably better to improve the design of your application than to shift the
> problem to asking for "yet another feature" of the RTOS!
> 
> In our robot control framework (orocos.org) that works as a user space
> application on top of Xenomai we have implemented lock-free data exchange,
> which does'nt require involvement of the scheduler at all, and which
> improves the timing performance of the highest priority thread. It might be
> useful for you to discuss this implementation aspect on the orocos
> developers mailinglist...
> 
> Herman Bruyninckx
> 


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] Xenomai and futexes - Native API optimized for

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2120 bytes --]

M. Koehrer wrote:
> Hi Herman and Jan,
> 
> thanks for your replies.
> I agree that is the best to do an approach that avoids critical sections.
> However, we have here the typical "historical heritage". Parts of this code have
> been ported from another system (where they worked with cli/sti).
> Other parts are generated from a third party code generator.
> This code generator (which is not in my hand) is designed primarily
> for offline simulation within a single task.
> Thus, the guys that programmed that, did not care about multi thread programming.
> The programs write the results into output variables.
> To avoid critical section I have now to do either
> an additional copy (also quite expensive) of the output data to a location 
> where I can work without any risk of being interrupted or I lock the access
>  to the data.
> That means either I do lots of additional copy actions (also quite expensive) or I work
> with the critical region.
> 
> Thus for me it is not an either - or - option.
> I can not optimize the code to be 100% free of critical sections (however it can be done in
> some parts..). That means I still have to count on the IPC provided by the OS.

OK, seeing your motivation, I would now like to roughly sketch to issue
I see about futexes for Xenomai:

You need user space and kernel space working on the same piece of memory
atomically AND you need to get the contention case right. The latter is
tricky because Xenomai does -for various reasons like robustness- the
sequence "try to acquire, fail, enqueue waiter, suspend" under one
common lock. As with futexes that "try to acquire" may cause page faults
(in buggy programs), you need to redesign things.

I have no model for such a redesign at hand (nor would I like to think
deeper and detract myself from other pending stuff), but my feeling is
that it will be at least, well, challenging. Looking at kernel futex
code and watching its pending issues, you may see that this is actually
not so much Xenomai-specific.

Anyway, go wild people, maybe I'm just too pessimistic. :)

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] Xenomai and futexes - Native API optimized for user space only applications

[Philippe Gerum]

On Wed, 2007-05-09 at 12:52 +0200, M. Koehrer wrote:
> Hi everybody,
> 
> I am using Xenomai for a high performance real time simulation system. 
> All of the simulation code is executed within user space. One application is running that consists
> of several Xenomai real time threads.
> For performance reasons I am always using the latest PC technology available (which is currently
> Pentium D or Core2Duo PCs, we are thinking even of using quad-core CPUs).
> There has to be a kind of thread synchronisation e.g. when accessing shared data.
> Hardware I/O is done via rtnet or via PCI I/O boards that work in user space aswell (PCI memory
> mapped into the user space).
> This is done e.g. by using semaphores, mutexes et.c
> I like the Xenomai-native skin as it provides a very clear API that is easy to use.
> However, for a user space only application it is a performance killer, that all API calls
> lead to a mode switch from user to kernel space. Each API call takes about 1-2 microseconds (us)
> on my PC which is really expensive.
> Especially when inter process communication is used to protect the access to shared data
> it is mostly the case that the calling thread does not have to wait. In this situation there is no
> need for a context switch. The API call did not lead to a rescheduling of the available tasks.
> And for this the required 1-2 us do really hurt.
> 
> Thus my question/proposal is if there is a plan to use a "variant" of the native API that is optimized for
> user space only applications. In this case e.g. futexes can be used. If there is a need to
> reschedule to another task it is fine to "invest" the 2us but it can be avoided mostly which should
> increase the overall performance dramatically.
> This would lead to a library where a big part of the functionality is handled directly in the library 
> (in user space). Currently the skin passes the (user space) API call via a Xenomai System call 
> to the kernel space to execute there the actual functionality.
> 
> Thanks for any feedback on this proposal.

Yes, this is the way to go. It's the kind of rework introduced by the
NPTL for the glibc, and there is no reason to pay the kernel/user space
transition when no contention exists on the synch object.

I'm not sure the cost is as much as 2 us, unless you don't use the
sysenter/sysexit protocol for syscalls Xenomai manages properly
(--enable-x86-sep). It seems you are going through the ancient 0x80
exception vector, but still, I agree that the point remains: no
contention should mean no transition to kernel.

We could not use futexes, because we don't want to depend on the vanilla
infrastructure, which would lead to unbounded latencies. The point is
about properly sharing a piece of data between both spaces for each
synch object, and have some user-space accessible atomic ops to operate
on them. That's the simplest part, the other, more complex one is to
invest the time needed to achieve this (for all archs that may run this
way, i.e. not all archs Xenomai supports do have atomic ops available
from user-space, but that's not the majority though).

> 
> Regards
> 
> Mathias
> 
> 
> 
> 
> 
-- 
Philippe.

Re: [Xenomai-help] Xenomai and futexes - Native API optimized for user space only applications

[Gilles Chanteperdrix]

Philippe Gerum wrote:
> On Wed, 2007-05-09 at 12:52 +0200, M. Koehrer wrote:
> 
>>Hi everybody,
>>
>>I am using Xenomai for a high performance real time simulation system. 
>>All of the simulation code is executed within user space. One application is running that consists
>>of several Xenomai real time threads.
>>For performance reasons I am always using the latest PC technology available (which is currently
>>Pentium D or Core2Duo PCs, we are thinking even of using quad-core CPUs).
>>There has to be a kind of thread synchronisation e.g. when accessing shared data.
>>Hardware I/O is done via rtnet or via PCI I/O boards that work in user space aswell (PCI memory
>>mapped into the user space).
>>This is done e.g. by using semaphores, mutexes et.c
>>I like the Xenomai-native skin as it provides a very clear API that is easy to use.
>>However, for a user space only application it is a performance killer, that all API calls
>>lead to a mode switch from user to kernel space. Each API call takes about 1-2 microseconds (us)
>>on my PC which is really expensive.
>>Especially when inter process communication is used to protect the access to shared data
>>it is mostly the case that the calling thread does not have to wait. In this situation there is no
>>need for a context switch. The API call did not lead to a rescheduling of the available tasks.
>>And for this the required 1-2 us do really hurt.
>>
>>Thus my question/proposal is if there is a plan to use a "variant" of the native API that is optimized for
>>user space only applications. In this case e.g. futexes can be used. If there is a need to
>>reschedule to another task it is fine to "invest" the 2us but it can be avoided mostly which should
>>increase the overall performance dramatically.
>>This would lead to a library where a big part of the functionality is handled directly in the library 
>>(in user space). Currently the skin passes the (user space) API call via a Xenomai System call 
>>to the kernel space to execute there the actual functionality.
>>
>>Thanks for any feedback on this proposal.
> 
> 
> Yes, this is the way to go. It's the kind of rework introduced by the
> NPTL for the glibc, and there is no reason to pay the kernel/user space
> transition when no contention exists on the synch object.
> 
> I'm not sure the cost is as much as 2 us, unless you don't use the
> sysenter/sysexit protocol for syscalls Xenomai manages properly
> (--enable-x86-sep). It seems you are going through the ancient 0x80
> exception vector, but still, I agree that the point remains: no
> contention should mean no transition to kernel.
> 
> We could not use futexes, because we don't want to depend on the vanilla
> infrastructure, which would lead to unbounded latencies. The point is
> about properly sharing a piece of data between both spaces for each
> synch object,

We could have a "sync object area" where all atomic counters would live,
that would be mapped into real-time processes address space, the sync
object would then only contain an index in this area. This would mean
that the number of sync objects is limited, but it is a restriction far
more acceptable than to allocate and share a full page by sync object.


> and have some user-space accessible atomic ops to operate
> on them. That's the simplest part, the other, more complex one is to
> invest the time needed to achieve this (for all archs that may run this
> way, i.e. not all archs Xenomai supports do have atomic ops available
> from user-space, but that's not the majority though).

For all archs, and for all skins.

-- 
                                                 Gilles Chanteperdrix

Re: [Xenomai-help] Xenomai and futexes - Native API optimized for

[Herman Bruyninckx]

On Wed, 9 May 2007, M. Koehrer wrote:

> thanks for your replies.
> I agree that is the best to do an approach that avoids critical sections.
> However, we have here the typical "historical heritage".

Nevertheless, I repeat my pledge _not_ to put new features in Xenomai
because of bad programming in legacy code! Every new feature will
automatically be considered by newcomers as good features, and that's _not_
the case! In my opinion, a good RTOS is one with a minimum of features, but
with a maximum of good application templates. (Call it "software patterns"
if you like.)

> That means I still have to count on the IPC provided by the OS.

I understand, but then I can't support your "complaint" about the Xenomai
features not being computationally optimal :-) _You_ have a bad legacy code,
so _you_ should pay the price for it, not the RTOS development or
maintainability...

Herman

Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm

Re: [Xenomai-help] Xenomai and futexes - Native API optimized

[M. Koehrer]

Hi Philippe,
> 
> I'm not sure the cost is as much as 2 us, unless you don't use the
> sysenter/sysexit protocol for syscalls Xenomai manages properly
> (--enable-x86-sep). It seems you are going through the ancient 0x80
> exception vector, but still, I agree that the point remains: no
> contention should mean no transition to kernel.
> 
I did some measurements to get precise values here.
For this I used the following Xenomai program that accesses a semaphore twice
within a loop (p followed by v).
---------------------------------
#include <native/task.h>
#include <native/sem.h>

RT_TASK taska_desc;
RT_SEM sem;

void mytaska(void *cookie)
{
    int i;
    RTIME tim1, tim2;
    #define LOOPS 100000

    tim1 = rt_timer_read();
    for (i=0; i < LOOPS; i++)
    {
        rt_sem_p(&sem, TM_INFINITE);
        rt_sem_v(&sem);
    }
    tim2 = rt_timer_read();
    printf("delta is %llu per step: %llu\n",
            tim2-tim1,
            (tim2-tim1)/ LOOPS);
}

int main(void)
{
    mlockall(MCL_CURRENT|MCL_FUTURE);
    rt_sem_create(&sem, "mysem", 1 ,0);
    rt_task_create(&taska_desc, "mytaska", 0, 81, T_JOINABLE);
    rt_task_start(&taska_desc, &mytaska, NULL);
    rt_task_join(&taska_desc);
    rt_sem_delete(&sem);
    return 0;
}
-----------------------------
I measured the following runtime:
For configuring Xenomai without any option:  4.8us per step
Configuring Xenomai with --enable-x86-sep: 4.5us per step.

I ran this experiment on a 3.2 GHz Pentium D on a server main board with
E7230 chipset.
Xenomai 2.3.1, kernel 2.6.20.4, SMP
Thus the performance here is not really excellent (as there is no need to do
a task switch.

I agree with all the comments that recommended to improve the design first before
trying to improve the OS performance. 
However, if there is (RT)OS, I expect it to be as fast as possible.
Because, even if I have a "perfect" design, I need the OS - otherwise I would just
live without the OS. 
A better design could help to reduce OS calls but it can not avoid it completely.
Also, a lock less designs might lead to additional data copies which is also a known 
performance killer. So I think this is a decision that has to be made on base of the concrete 
project. General rules do not help here.

Thanks for all comments!

Regards

Mathias





-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] Xenomai and futexes - Native API optimized

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1206 bytes --]

M. Koehrer wrote:
> ...
> I measured the following runtime:
> For configuring Xenomai without any option:  4.8us per step
> Configuring Xenomai with --enable-x86-sep: 4.5us per step.
> 
> I ran this experiment on a 3.2 GHz Pentium D on a server main board with
> E7230 chipset.
> Xenomai 2.3.1, kernel 2.6.20.4, SMP
> Thus the performance here is not really excellent (as there is no need to do
> a task switch.

Mind to run oprofile on this setup to see where costs primarily come from?

> 
> I agree with all the comments that recommended to improve the design first before
> trying to improve the OS performance. 
> However, if there is (RT)OS, I expect it to be as fast as possible.

And there it is again, the common misunderstanding: RTOSes are not
GPOSes, only faster. They provide services optimised for predictable and
low (in that order) worst-case performance.

So adding optimisations for whatever average case must not hurt those
goals significantly. And that often means your overall performance is
even worse than with an GPOS. No excuses for keeping Xenomai unoptimised
if it can be tuned smartly, just an explanation why we may not help in
every case.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] Xenomai and futexes - Native API optimized

[Daniel Schnell]

Hi,

I find the results interesting.

running your program under a MPC5200B, 396 MHz yields slightly better results: 3847 ns ~ 3,8 us.
For the rt_sem_v() operation alone that is 1778 ns ~1,7 us ==> rt_sem_p(): 2069 ns ~2,1 us.

Indeed for a ctx switch less operation this is a _very_ long time. It would be interesting to know how the figures are for kernel based operations.

I have not measured mutexes so far, but it is probable that the figures are similar. If so, this is really something one should try to improve, because one has to "pay" a lot for the relatively unlikely event of a reschedule. And mutexes are and will/should be used a lot.

I have to support Mathias here: it is not so easy as blaming bad design of the application or the used tooling. One often has dependencies on legacy code / tools / external libraries. Especially so in industrial environment, where standard protocols and interfaces are used, just think about a full featured CanOpen protocol stack for typically a couple of 1000 €. One simply cannot create a software "master plan" and fix all possible design flaws in those external dependent components. Especially not so in external binary only libraries.

We are at the moment underway to find out the specifications of our Xenomai platform, i.e. max. ctx switch times, performance of the POSIX skin, etc.

I will post the results and the source code if we are finished.

Best regards,

-- 
Daniel Schnell                   | daniel.schnell@marel.com
Hugbúnaðargerð                   | www.marel.com
 

-----Original Message-----
From: xenomai-help-bounces@gna.org [mailto:xenomai-help-bounces@gna.org] On Behalf Of M. Koehrer
Sent: 10. maí 2007 08:31
To: rpm@xenomai.org; mathias_koehrer@arcor.de
Cc: xenomai-help@gna.org
Subject: Re: [Xenomai-help] Xenomai and futexes - Native API optimized

Hi Philippe,
> 
> I'm not sure the cost is as much as 2 us, unless you don't use the 
> sysenter/sysexit protocol for syscalls Xenomai manages properly 
> (--enable-x86-sep). It seems you are going through the ancient 0x80 
> exception vector, but still, I agree that the point remains: no 
> contention should mean no transition to kernel.
> 
I did some measurements to get precise values here.
For this I used the following Xenomai program that accesses a semaphore twice within a loop (p followed by v).
---------------------------------
#include <native/task.h>
#include <native/sem.h>

RT_TASK taska_desc;
RT_SEM sem;

void mytaska(void *cookie)
{
    int i;
    RTIME tim1, tim2;
    #define LOOPS 100000

    tim1 = rt_timer_read();
    for (i=0; i < LOOPS; i++)
    {
        rt_sem_p(&sem, TM_INFINITE);
        rt_sem_v(&sem);
    }
    tim2 = rt_timer_read();
    printf("delta is %llu per step: %llu\n",
            tim2-tim1,
            (tim2-tim1)/ LOOPS);
}

int main(void)
{
    mlockall(MCL_CURRENT|MCL_FUTURE);
    rt_sem_create(&sem, "mysem", 1 ,0);
    rt_task_create(&taska_desc, "mytaska", 0, 81, T_JOINABLE);
    rt_task_start(&taska_desc, &mytaska, NULL);
    rt_task_join(&taska_desc);
    rt_sem_delete(&sem);
    return 0;
}
-----------------------------
I measured the following runtime:
For configuring Xenomai without any option:  4.8us per step Configuring Xenomai with --enable-x86-sep: 4.5us per step.

I ran this experiment on a 3.2 GHz Pentium D on a server main board with E7230 chipset.
Xenomai 2.3.1, kernel 2.6.20.4, SMP
Thus the performance here is not really excellent (as there is no need to do a task switch.

I agree with all the comments that recommended to improve the design first before trying to improve the OS performance. 
However, if there is (RT)OS, I expect it to be as fast as possible.
Because, even if I have a "perfect" design, I need the OS - otherwise I would just live without the OS. 
A better design could help to reduce OS calls but it can not avoid it completely.
Also, a lock less designs might lead to additional data copies which is also a known performance killer. So I think this is a decision that has to be made on base of the concrete project. General rules do not help here.

Thanks for all comments!

Regards

Mathias





--
Mathias Koehrer
mathias_koehrer@arcor.de


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

_______________________________________________
Xenomai-help mailing list
Xenomai-help@gna.org
https://mail.gna.org/listinfo/xenomai-help

Re: [Xenomai-help] Xenomai and futexes - Native API optimized

[M. Koehrer]

Hi Jan,

it is the first time I use oprofile, I hope I have here results that are significant...
I wrote a bash script  to start everything:
--------------------
#!/bin/bash
opcontrol --start
./performance
opcontrol --stop
opreport
--------------------
My real time test application is called "performance".
Here is the output:
------------------------------------------------------------
Profiler running.
delta is 2023521331 per step: 5058
Stopping profiling.
CPU: P4 / Xeon, speed 3192.18 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
GLOBAL_POWER_E...|
  samples|      %|
------------------
    38103 76.4368 vmlinux
     2659  5.3341 xeno_nucleus
     2327  4.6681 libnative.so.0.0.0
     2078  4.1686 bash
        GLOBAL_POWER_E...|
          samples|      %|
        ------------------
             2069 99.5669 bash
                3  0.1444 anon (tgid:4769 range:0xb7fea000-0xb7feb000)
                2  0.0962 anon (tgid:22703 range:0xb7fea000-0xb7feb000)
                2  0.0962 anon (tgid:4883 range:0xb7fea000-0xb7feb000)
                1  0.0481 anon (tgid:1883 range:0xb7fea000-0xb7feb000)
                1  0.0481 anon (tgid:4942 range:0xb7fea000-0xb7feb000)
     1587  3.1836 xeno_native
     1514  3.0372 libc-2.3.6.so
      907  1.8195 ld-2.3.6.so
      317  0.6359 oprofiled
        GLOBAL_POWER_E...|
          samples|      %|
        ------------------
              315 99.3691 oprofiled
                2  0.6309 anon (tgid:1855 range:0xb7fea000-0xb7feb000)
      142  0.2849 performance
        GLOBAL_POWER_E...|
          samples|      %|
        ------------------
              130 91.5493 anon (tgid:22701 range:0xb7fea000-0xb7feb000)
               12  8.4507 performance
       91  0.1826 oprofile
       33  0.0662 gawk
       25  0.0502 e1000
       24  0.0481 grep
        7  0.0140 ls
        4  0.0080 cat
        4  0.0080 libncurses.so.5.5
        3  0.0060 libpopt.so.0.0.0
        3  0.0060 libselinux.so.1
        3  0.0060 libsepol.so.1
        3  0.0060 libdl-2.3.6.so
        3  0.0060 libpthread-2.3.6.so
        2  0.0040 mkdir
        2  0.0040 ophelp
        1  0.0020 lsmod
        1  0.0020 mktemp
        1  0.0020 libnss_compat-2.3.6.so
        1  0.0020 libnss_files-2.3.6.so
        1  0.0020 dirname
        1  0.0020 id
        1  0.0020 tr
        1  0.0020 in.telnetd
------------------------------------------


> 
> Mind to run oprofile on this setup to see where costs primarily come from?
> 

Does this help??

Regards

Mathias

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] Xenomai and futexes - Native API optimized

[M. Koehrer]

Hi Jan,

here is the (truncated) output of opreport -l (which seems to be more useful...).

Regards

Mathias
------------------
Profiler running.
delta is 2030247695 per step: 5075
Stopping profiling.
CPU: P4 / Xeon, speed 3192.18 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
samples  %        image name               app name                 symbol name
27423    64.2074  vmlinux                  vmlinux                  __ipipe_hard_cpuid
2309      5.4062  xeno_nucleus             xeno_nucleus             (no symbols)
1489      3.4863  bash                     bash                     (no symbols)
1226      2.8705  vmlinux                  vmlinux                  sysenter_past_esp
1209      2.8307  xeno_native              xeno_native              (no symbols)
1091      2.5544  libc-2.3.6.so            libc-2.3.6.so            (no symbols)
792       1.8544  vmlinux                  vmlinux                  __ipipe_syscall_root
700       1.6390  vmlinux                  vmlinux                  __ipipe_dispatch_event
581       1.3603  vmlinux                  vmlinux                  page_fault
316       0.7399  vmlinux                  vmlinux                  __ipipe_stall_root
264       0.6181  vmlinux                  vmlinux                  do_wp_page
232       0.5432  oprofiled                oprofiled                (no symbols)
227       0.5315  vmlinux                  vmlinux                  __ipipe_test_and_stall_root
177       0.4144  ld-2.3.6.so              ld-2.3.6.so              do_lookup_x
157       0.3676  vmlinux                  vmlinux                  find_get_page
156       0.3653  vmlinux                  vmlinux                  __copy_from_user_ll_nozero
156       0.3653  vmlinux                  vmlinux                  __handle_mm_fault
128       0.2997  performance              performance              mytaska
122       0.2856  ld-2.3.6.so              ld-2.3.6.so              strcmp
106       0.2482  vmlinux                  vmlinux                  unmap_vmas
105       0.2458  vmlinux                  vmlinux                  __ipipe_handle_exception
97        0.2271  vmlinux                  vmlinux                  get_page_from_freelist
92        0.2154  vmlinux                  vmlinux                  search_by_key
86        0.2014  vmlinux                  vmlinux                  page_remove_rmap
85        0.1990  vmlinux                  vmlinux                  flush_tlb_page
82        0.1920  vmlinux                  vmlinux                  timer_interrupt
78        0.1826  vmlinux                  vmlinux                  filemap_nopage
77        0.1803  vmlinux                  vmlinux                  __ipipe_unstall_root
75        0.1756  vmlinux                  vmlinux                  release_pages
73        0.1709  oprofile                 oprofile                 (no symbols)
66        0.1545  vmlinux                  vmlinux                  __ipipe_restore_pipeline_head
64        0.1498  vmlinux                  vmlinux                  mwait_idle_with_hints
62        0.1452  ld-2.3.6.so              ld-2.3.6.so              _dl_relocate_object
60        0.1405  vmlinux                  vmlinux                  copy_page_range
60        0.1405  vmlinux                  vmlinux                  find_vma
56        0.1311  vmlinux                  vmlinux                  hrtimer_run_queues
52        0.1218  vmlinux                  vmlinux                  __ipipe_test_root
52        0.1218  vmlinux                  vmlinux                  up_read
50        0.1171  vmlinux                  vmlinux                  __ipipe_unstall_iret_root
49        0.1147  vmlinux                  vmlinux                  do_page_fault
47        0.1100  e1000                    e1000                    (no symbols)
47        0.1100  vmlinux                  vmlinux                  down_read_trylock
46        0.1077  vmlinux                  vmlinux                  error_code
40        0.0937  vmlinux                  vmlinux                  __ipipe_sync_stage
---------------------------
 


> 
> > 
> > Mind to run oprofile on this setup to see where costs primarily come
> from?
> > 


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  39,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] Xenomai and futexes - Native API optimized

[Philippe Gerum]

On Thu, 2007-05-10 at 09:46 +0000, Daniel Schnell wrote:
> If so, this is really something one should try to improve, because one has to "pay" a lot
>  for the relatively unlikely event of a reschedule. And mutexes are and will/should be used a lot.
> 
> I have to support Mathias here: it is not so easy as blaming bad design of the application or the used tooling.
> One often has dependencies on legacy code / tools / external libraries.

Ack. Xenomai is very much about helping people to migrate from legacy RT
environments to Linux-based systems; this is why Xenomai has a skin
layer instead of a single wired-in interface in the first place. In that
sense, we _must_ accept the fact that such code needs to be ported with
the fewest changes that make the transition possible, basically because
even as mis-designed or sloppy as it could be, such code often has one
single very desirable property: it _works in the field_ (damnit! (c)),
and an awful lot of effort has likely been put to fix the real-world
issues which have popped during the application lifetime no one wants to
start again, and any knowledge about those bugs and how they have been
fixed has often vanished many moons ago.

To sum up, I have no problem trying hard to make some ugly legacy code
shine by mean of a smart implementation on our side, that's usual
business after all.

This is not to say that we should plumb any number of evil core
interfaces into Xenomai, or tweak the existing ones, just for making new
things as silly or ugly as they might have been in the legacy context.
But, improving the performance of mutual exclusion would _not_ impact
any Xenomai interface, but only the innards of some skins, for the best
in terms of performance. So, FWIW, I agree, we must improve this when
time allows, and yes, it should probably be high enough priority-wise in
the TODO list, especially if measurements confirm the problem.

I see two distinct improvements we could achieve:

- optimize our syscall path, between user-space and the nucleus. This
would have a positive impact on every syscall from every skin. My
feeling is that I-cache penalty is way too high presently when
dispatching syscall events within the Adeos layer. We must do so while
keeping the pipeline scheme so we could still downgrade syscalls from
the Xenomai domain to the Linux one when needed.

- implement the fast acquisition path for core Xenomai mutexes (to be
defined) in the non-contended case, using a piece of pinned memory
shared between kernel and user spaces.

-- 
Philippe.

Re: [Xenomai-help] Xenomai and futexes - Native API optimized

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

M. Koehrer wrote:
> Hi Jan,
> 
> here is the (truncated) output of opreport -l (which seems to be more useful...).

Are you sure you predominately measured the semaphore scenario, not also
some things that happens before and after (or during) that test? Make
sure the test loop runs longer than setup/cleanup.

> 
> Regards
> 
> Mathias
> ------------------
> Profiler running.
> delta is 2030247695 per step: 5075
> Stopping profiling.
> CPU: P4 / Xeon, speed 3192.18 MHz (estimated)
> Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
> samples  %        image name               app name                 symbol name
> 27423    64.2074  vmlinux                  vmlinux                  __ipipe_hard_cpuid

Mmpf, that's heavy. Can someone comment on this? Is the related hardware
access that costly or are we calling it too often?

> 2309      5.4062  xeno_nucleus             xeno_nucleus             (no symbols)

We are lacking module symbols here. Dunno if one can teach them to
oprofile (likely somehow), but the easiest approach would be to compile
xeno_native and nucleus into the kernel.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]