[Xenomai-help] overhead

[Xenomai-help] overhead

[Steven Seeger]

[-- Attachment #1: Type: text/plain, Size: 233 bytes --]

Is it normal for Xenomai to have significant overhead in userspace
compared to older software running RTAI with all realtime threads in
kernel modules? My application makes heavy use of inb/outb with iopl().

 

Steven

 


[-- Attachment #2: Type: text/html, Size: 1870 bytes --]

Re: [Xenomai-help] overhead

[Jan Kiszka]

Steven Seeger wrote:
> Is it normal for Xenomai to have significant overhead in userspace
> compared to older software running RTAI with all realtime threads in
> kernel modules? My application makes heavy use of inb/outb with iopl().

Significant overhead in user land can come from significant task
switches or a significant number of syscalls. But direct I/O should not
bite you. Any suspicious number of faults under /proc/xenomai/faults?

How did you measure the overhead in both systems?

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] overhead

[Steven Seeger]

The overhead was measured by only comparing cpu usage between tasks from
the statistics collection stuff. There are some userland tasks running
at 8khz, so apparently this is a problem for Xenomai on this board. I
had no problem at all with these tasks in kernel space under rtai. 

There are no faults at all under /proc/xenomai/faults.

The main way I can tell something is wrong is that if my three fast
tasks are running (8 khz in kernel space, 8 khz in userspace, and 50 hz
in userspace) then a lower priority task will timeout and trip a
watchdog. If I remove the IO routines from the 50hz task, then the
others can still run.

Steven


> -----Original Message-----
> From: Jan Kiszka [mailto:jan.kiszka@domain.hid]
> Sent: Monday, February 11, 2008 6:55 AM
> To: Steven Seeger
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai-help] overhead
> 
> Steven Seeger wrote:
> > Is it normal for Xenomai to have significant overhead in userspace
> > compared to older software running RTAI with all realtime threads in
> > kernel modules? My application makes heavy use of inb/outb with
iopl().
> 
> Significant overhead in user land can come from significant task
> switches or a significant number of syscalls. But direct I/O should
not
> bite you. Any suspicious number of faults under /proc/xenomai/faults?
> 
> How did you measure the overhead in both systems?
> 
> Jan
> 
> --
> Siemens AG, Corporate Technology, CT SE 2
> Corporate Competence Center Embedded Linux

Re: [Xenomai-help] overhead

[Gilles Chanteperdrix]

On Mon, Feb 11, 2008 at 4:20 PM, Steven Seeger
<sseeger@domain.hid> wrote:
> The overhead was measured by only comparing cpu usage between tasks from
>  the statistics collection stuff. There are some userland tasks running
>  at 8khz, so apparently this is a problem for Xenomai on this board. I
>  had no problem at all with these tasks in kernel space under rtai.

user-space adds some overhead compared to kernel-space. Enabling the
T_FPU bis for a task which does not need also adds the useless
overhead of saving/restoring FPU upon context switch.

-- 
                                               Gilles Chanteperdrix

Re: [Xenomai-help] overhead

[Jan Kiszka]

Steven Seeger wrote:
> The overhead was measured by only comparing cpu usage between tasks from
> the statistics collection stuff. There are some userland tasks running
> at 8khz, so apparently this is a problem for Xenomai on this board. I
> had no problem at all with these tasks in kernel space under rtai. 
> 
> There are no faults at all under /proc/xenomai/faults.
> 
> The main way I can tell something is wrong is that if my three fast
> tasks are running (8 khz in kernel space, 8 khz in userspace, and 50 hz
> in userspace) then a lower priority task will timeout and trip a
> watchdog. If I remove the IO routines from the 50hz task, then the
> others can still run.

I/O does not come for free (specifically on ISA HW - if that is the case
for you). But if you think that this makes a difference here, what about
measuring both kernel- and userland-based I/O accesses in a loop and
compare the runtime? This would also allow you to quantify the impact -
regardless if there is a difference or not.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] overhead

[Steven Seeger]

Well one of my 8khz tasks in kernel space seems to require T_FPU despite
me never using it. I've been able to reproduce my bug within a minute
several times without that flag, but if the flag is set, it can run for
an hour without a single value being corrupted into NaN. 

I also am getting a lot of page faults with ipipe_handle_irq() in the
back trace. I'm also noticing if I have T_WARNSW on for a primary mode
only thread, it will receive the signal if I create a Linux-only thread
with pthread_create() and that's just weird.

I don't think the problem is IO time. It's definitely context switching
and things. IRQ0 handler uses up to 20% of the cpu, and that just seems
wrong. It gets worse if I try the other scalability options (tree or
hash.) It uses a lot of CPU time even when the tasks are delayed on
conditional variables. I don't have periodic timing on, either.

I never used to notice such high CPU usage with older versions of
Xenomai. I even had an application written that ran all these threads
and didn't have a problem with it. 

I'll hopefully find some time soon to write isolated tests and try to
figure out what's going on. The page fault problems (there are more than
the one I posted earlier) are really getting annoying, too.

Sadly my contractor is on the verge of scrapping Xenomai for their
project because of these problems. :( I really believe in it, but I'm
basically working two jobs and don't have all the time I wish I did to
help test.

And Paul, it is only on x86. 

Steven

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Monday, February 11, 2008 9:10 AM
> To: Steven Seeger
> Cc: Jan Kiszka; xenomai@xenomai.org
> Subject: Re: [Xenomai-help] overhead
> 
> On Mon, Feb 11, 2008 at 4:20 PM, Steven Seeger
> <sseeger@domain.hid> wrote:
> > The overhead was measured by only comparing cpu usage between tasks
from
> >  the statistics collection stuff. There are some userland tasks
running
> >  at 8khz, so apparently this is a problem for Xenomai on this board.
I
> >  had no problem at all with these tasks in kernel space under rtai.
> 
> user-space adds some overhead compared to kernel-space. Enabling the
> T_FPU bis for a task which does not need also adds the useless
> overhead of saving/restoring FPU upon context switch.
> 
> --
>                                                Gilles Chanteperdrix

Re: [Xenomai-help] overhead

[Jan Kiszka]

Steven Seeger wrote:
> Well one of my 8khz tasks in kernel space seems to require T_FPU despite
> me never using it. I've been able to reproduce my bug within a minute
> several times without that flag, but if the flag is set, it can run for
> an hour without a single value being corrupted into NaN. 
> 
> I also am getting a lot of page faults with ipipe_handle_irq() in the
> back trace. I'm also noticing if I have T_WARNSW on for a primary mode
> only thread, it will receive the signal if I create a Linux-only thread
> with pthread_create() and that's just weird.

Hmm? You call pthread_create() from the primary-mode-only thread? Or
from some third, unrelated one?

> 
> I don't think the problem is IO time. It's definitely context switching
> and things. IRQ0 handler uses up to 20% of the cpu, and that just seems
> wrong. It gets worse if I try the other scalability options (tree or
> hash.) It uses a lot of CPU time even when the tasks are delayed on
> conditional variables. I don't have periodic timing on, either.

What CPU configuration are you using / do you have to use with your
board? If your selection enforces PIC/PIT, that could explain noticeable
overhead of the timer. Do you have TSC support on your platform? Is it
enabled and usable? Check the .config (or post it), have a look at
/proc/xenomai/timer (latest versions should report clock/timer
settings), check the TSC clock stability with clocktest from the testsuite.

> 
> I never used to notice such high CPU usage with older versions of
> Xenomai. I even had an application written that ran all these threads
> and didn't have a problem with it. 

Older means before 2.4? Or even older than 2.3?

> 
> I'll hopefully find some time soon to write isolated tests and try to
> figure out what's going on. The page fault problems (there are more than
> the one I posted earlier) are really getting annoying, too.
> 
> Sadly my contractor is on the verge of scrapping Xenomai for their
> project because of these problems. :( I really believe in it, but I'm
> basically working two jobs and don't have all the time I wish I did to
> help test.

Do they have an alternative ready at hand? Blindly swapping the problems
rarely helps, first understand, then decide what to do (I know I'm
barking at the wrong tree... ;) ).

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] overhead

[Philippe Gerum]

Steven Seeger wrote:
> Well one of my 8khz tasks in kernel space seems to require T_FPU despite
> me never using it. I've been able to reproduce my bug within a minute
> several times without that flag, but if the flag is set, it can run for
> an hour without a single value being corrupted into NaN. 
> 
> I also am getting a lot of page faults with ipipe_handle_irq() in the
> back trace.

Send those backtraces, please, and a disassembly of your running kernel
privately to me.

 I'm also noticing if I have T_WARNSW on for a primary mode
> only thread, it will receive the signal if I create a Linux-only thread
> with pthread_create() and that's just weird.
>

Sorry, I just don't understand where the problem is, or maybe I did not
understand what you are exactly doing/expecting here.

> I don't think the problem is IO time. It's definitely context switching
> and things. IRQ0 handler uses up to 20% of the cpu, and that just seems
> wrong. It gets worse if I try the other scalability options (tree or
> hash.) It uses a lot of CPU time even when the tasks are delayed on
> conditional variables. I don't have periodic timing on, either.
> 

If you actually take faults on behalf of the tick timer handler, then no
wonder why the CPU figures explode.

> I never used to notice such high CPU usage with older versions of
> Xenomai.

Older versions, or older kernels?

 I even had an application written that ran all these threads
> and didn't have a problem with it. 
> 
> I'll hopefully find some time soon to write isolated tests and try to
> figure out what's going on. The page fault problems (there are more than
> the one I posted earlier) are really getting annoying, too.
> 
> Sadly my contractor is on the verge of scrapping Xenomai for their
> project because of these problems. :( I really believe in it, but I'm
> basically working two jobs and don't have all the time I wish I did to
> help test.
> 
> And Paul, it is only on x86. 
> 
> Steven
> 
>> -----Original Message-----
>> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
>> Sent: Monday, February 11, 2008 9:10 AM
>> To: Steven Seeger
>> Cc: Jan Kiszka; xenomai@xenomai.org
>> Subject: Re: [Xenomai-help] overhead
>>
>> On Mon, Feb 11, 2008 at 4:20 PM, Steven Seeger
>> <sseeger@domain.hid> wrote:
>>> The overhead was measured by only comparing cpu usage between tasks
> from
>>>  the statistics collection stuff. There are some userland tasks
> running
>>>  at 8khz, so apparently this is a problem for Xenomai on this board.
> I
>>>  had no problem at all with these tasks in kernel space under rtai.
>> user-space adds some overhead compared to kernel-space. Enabling the
>> T_FPU bis for a task which does not need also adds the useless
>> overhead of saving/restoring FPU upon context switch.
>>
>> --
>>                                                Gilles Chanteperdrix
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
> 


-- 
Philippe.

Re: [Xenomai-help] overhead

[Steven Seeger]

> Send those backtraces, please, and a disassembly of your running
kernel
> privately to me.

I don't have the backtraces because they'd occur and the system would
freeze. I wasn't using serial console because my application uses both
serial ports. I even noticed ipipe_ calls and 16554_ serial calls in the
same backtrace.

Next time one occurs, I will be sure to post it.

> Sorry, I just don't understand where the problem is, or maybe I did
not
> understand what you are exactly doing/expecting here.

I wanted to be warned if my main state thread ever went to secondary
domain, so I used T_WARNSW. However, I posted this condition wrong. The
SIGXCPU is caused by my calling assert() because my state thread had an
overrun, which it should not have. The system is doing nothing at this
point, but calling pthread_create() in a shadowed thread causes an
overrun. The shadowed thread is priority 1, and the state thread is
priority 50. The state thread runs at 100Hz so this should not be too
bad on the system.
 
> If you actually take faults on behalf of the tick timer handler, then
no
> wonder why the CPU figures explode.

I don't understand what you mean here?

> Older versions, or older kernels?

Xenomai 2.2 era. I think I used 2.6.18 or something. It was a long time
ago.
Steven

Re: [Xenomai-help] overhead

[Philippe Gerum]

Steven Seeger wrote:
>> Send those backtraces, please, and a disassembly of your running
> kernel
>> privately to me.
> 
> I don't have the backtraces because they'd occur and the system would
> freeze. I wasn't using serial console because my application uses both
> serial ports. I even noticed ipipe_ calls and 16554_ serial calls in the
> same backtrace.
> 
> Next time one occurs, I will be sure to post it.
> 

At least the topmost four/five lines would help. You will find
__ipipe_handle_irq within the call frame each time an interrupt is
taken, so you need to focus on which handler gets actually called. In
the case above, it seems a serial interrupt is involved, and what its
handler does might be the root of the issue.

>> Sorry, I just don't understand where the problem is, or maybe I did
> not
>> understand what you are exactly doing/expecting here.
> 
> I wanted to be warned if my main state thread ever went to secondary
> domain, so I used T_WARNSW. However, I posted this condition wrong. The
> SIGXCPU is caused by my calling assert() because my state thread had an
> overrun, which it should not have. The system is doing nothing at this
> point, but calling pthread_create() in a shadowed thread causes an
> overrun. The shadowed thread is priority 1, and the state thread is
> priority 50. The state thread runs at 100Hz so this should not be too
> bad on the system.
>

Creating threads should be kept out of real-time duties, it's just
unsafe in terms of latency (Btw, I hope you do reduce the default glibc
stacksize from 8Mb to something more sensible in a real-time application
context, given that mlockall is in effect).

>> If you actually take faults on behalf of the tick timer handler, then
> no
>> wonder why the CPU figures explode.
> 
> I don't understand what you mean here?
> 

ipipe_handle_irq will be present in every call frame on behalf of an
external or virtual interrupt context, therefore if you get a fault in
the interrupt handler like you reported lately, the time spent in the
fault recovery code will be charged to the interrupt context.

>> Older versions, or older kernels?
> 
> Xenomai 2.2 era. I think I used 2.6.18 or something. It was a long time
> ago.
> Steven
> 
> 


-- 
Philippe.

Re: [Xenomai-help] overhead

[Steven Seeger]

> At least the topmost four/five lines would help. You will find
> __ipipe_handle_irq within the call frame each time an interrupt is
> taken, so you need to focus on which handler gets actually called. In
> the case above, it seems a serial interrupt is involved, and what its
> handler does might be the root of the issue.

I'll run these tests tonight for you and post the results.

> Creating threads should be kept out of real-time duties, it's just
> unsafe in terms of latency (Btw, I hope you do reduce the default
glibc
> stacksize from 8Mb to something more sensible in a real-time
application
> context, given that mlockall is in effect).

I understand this, and this is something I always follow. However, this
is a linux only pthread not being created in a realtime context. The
only reason the thread is shadowed is so it can call a few things
occasionally to lock an RT_MUTEX. None of these mutexes are held at this
time, and the thread might as well be in secondary mode. (and probably
is.)

> ipipe_handle_irq will be present in every call frame on behalf of an
> external or virtual interrupt context, therefore if you get a fault in
> the interrupt handler like you reported lately, the time spent in the
> fault recovery code will be charged to the interrupt context.

Thanks for the explanation. I'll be sure to gather some of these
tonight. I sent you the only one I have privately with the kernel.

Steven