[Xenomai-help] in-kernel periodic task latencies caused by dohell

[Xenomai-help] in-kernel periodic task latencies caused by dohell

[Eric Eric]

[-- Attachment #1: Type: text/plain, Size: 1313 bytes --]

Hello, I've been doing some more Xenomai benchmarking lately and have come
across what seems to be strange behavior.  I'm running Xenomai 2.6.0 on a
Beagleboard XM with kernel 2.6.37.  Just running the in-kernel periodic
task benchmark (./latency -t1) alone results in an average of 2.7uS
latency.  What is interesting is the effect of various parts of dohell:

1) If I just run unmodified dohell with only the seconds argument, the
latency goes to about 17uS.

2) If I comment out all tests in dohell except for "dd if=/dev/zero
of=/dev/null &", average latency goes to 3.3uS (with top indicating about
98% CPU usage by dd).

3) If I comment out all tests in dohell except for "while :; do cat
/proc/interrupts; done > /dev/null 2>&1 &", average latency goes to about
24uS.

So it seems that trapping into the Linux kernel can have a drastic effect
on Xenomai real-time kernel latencies.  Why is this?  Shouldn't a
timer-driven runnable real-time Xenomai kernel task require the same
preemption latency regardless of whether Linux is busy or not?  And it's
not just that Linux processes are loading the CPU (this happens in case 2
with relatively little effect on latency), but that it's the act of
trapping into the kernel that causes the latency.  Any ideas why this is
the case?

Thank you very much.

- Eric

[-- Attachment #2: Type: text/html, Size: 1422 bytes --]

Re: [Xenomai-help] in-kernel periodic task latencies caused by dohell

[Gilles Chanteperdrix]

On 02/07/2012 01:01 AM, Eric Eric wrote:
> Hello, I've been doing some more Xenomai benchmarking lately and have come
> across what seems to be strange behavior.  I'm running Xenomai 2.6.0 on a
> Beagleboard XM with kernel 2.6.37.  Just running the in-kernel periodic
> task benchmark (./latency -t1) alone results in an average of 2.7uS
> latency.  What is interesting is the effect of various parts of dohell:
> 
> 1) If I just run unmodified dohell with only the seconds argument, the
> latency goes to about 17uS.
> 
> 2) If I comment out all tests in dohell except for "dd if=/dev/zero
> of=/dev/null &", average latency goes to 3.3uS (with top indicating about
> 98% CPU usage by dd).
> 
> 3) If I comment out all tests in dohell except for "while :; do cat
> /proc/interrupts; done > /dev/null 2>&1 &", average latency goes to about
> 24uS.
> 
> So it seems that trapping into the Linux kernel can have a drastic effect
> on Xenomai real-time kernel latencies. 

You are drawing conclusions a bit too fast. dd also causes a lot of
trapping into the kernel. If you are interested in knowing how the time
is spent, I suggest using the I-pipe tracer.

If you find an improvement of the I-pipe patch for your platform, we
will gladly welcome patches.

-- 
                                                                Gilles.

Re: [Xenomai-help] in-kernel periodic task latencies caused by dohell

[Eric Eric]

Thanks, Gilles.  I enabled ipipe tracing but am a bit confused by what
it's telling me.  What I would like to do is generate a trace during
the timed section of the benchmark, and not anywhere else.  Looking at
the output, it seems max and frozen contain traces of the longest path
to date.  It's not clear to me how this maps to my benchmark
latencies.  For example, I can sample max and/or frozen and add up the
latency column and see them seemingly randomly exceed each other
regardless of whether dohell is running.

Looking at http://www.xenomai.org/index.php/Xenomai:I-pipe_Tracer#API
I see there are calls like xntrace_user_start and xntrace_user_stop.
Is it possible to wrap the timed parts of the benchmarks with calls
like this to get a more informative trace?

Also, do you think the latency degradation may be due to the VIVT
cache on the ARM?  I'm specifically referring to additional latency
due to different address spaces when switching between Linux user-mode
and Xenomai kernel.

Thanks again.

- Eric

On Tue, Feb 7, 2012 at 10:22 AM, Gilles Chanteperdrix
<gilles.chanteperdrix@xenomai.org> wrote:
>
> On 02/07/2012 01:01 AM, Eric Eric wrote:
> > Hello, I've been doing some more Xenomai benchmarking lately and have come
> > across what seems to be strange behavior.  I'm running Xenomai 2.6.0 on a
> > Beagleboard XM with kernel 2.6.37.  Just running the in-kernel periodic
> > task benchmark (./latency -t1) alone results in an average of 2.7uS
> > latency.  What is interesting is the effect of various parts of dohell:
> >
> > 1) If I just run unmodified dohell with only the seconds argument, the
> > latency goes to about 17uS.
> >
> > 2) If I comment out all tests in dohell except for "dd if=/dev/zero
> > of=/dev/null &", average latency goes to 3.3uS (with top indicating about
> > 98% CPU usage by dd).
> >
> > 3) If I comment out all tests in dohell except for "while :; do cat
> > /proc/interrupts; done > /dev/null 2>&1 &", average latency goes to about
> > 24uS.
> >
> > So it seems that trapping into the Linux kernel can have a drastic effect
> > on Xenomai real-time kernel latencies.
>
> You are drawing conclusions a bit too fast. dd also causes a lot of
> trapping into the kernel. If you are interested in knowing how the time
> is spent, I suggest using the I-pipe tracer.
>
> If you find an improvement of the I-pipe patch for your platform, we
> will gladly welcome patches.
>
> --
>                                                                Gilles.

Re: [Xenomai-help] in-kernel periodic task latencies caused by dohell

[Gilles Chanteperdrix]

On 02/08/2012 01:25 AM, Eric Eric wrote:
> Thanks, Gilles.  I enabled ipipe tracing but am a bit confused by what
> it's telling me.  What I would like to do is generate a trace during
> the timed section of the benchmark, and not anywhere else.  Looking at
> the output, it seems max and frozen contain traces of the longest path
> to date.  It's not clear to me how this maps to my benchmark
> latencies.  For example, I can sample max and/or frozen and add up the
> latency column and see them seemingly randomly exceed each other
> regardless of whether dohell is running.

The idea is to run latency with the -f option so that when you hit the
maximum with whatever load you are interested in, it is recorded. Be
aware though, that the tracer adds some overhead, so for instance your
24us latency may well turn into a 50us latency when run with the tracer.

> 
> Looking at http://www.xenomai.org/index.php/Xenomai:I-pipe_Tracer#API
> I see there are calls like xntrace_user_start and xntrace_user_stop.
> Is it possible to wrap the timed parts of the benchmarks with calls
> like this to get a more informative trace?

I do not really understand your problem. If you know that you hit the
maximum latency while running the cat /proc/interrupts loop, then
latency -f should do just fine.

> 
> Also, do you think the latency degradation may be due to the VIVT
> cache on the ARM?  I'm specifically referring to additional latency
> due to different address spaces when switching between Linux user-mode
> and Xenomai kernel.
> 

As I said, the fact that the maximum happens with cat /proc/interrupts
probably has nothing to do with kernel/user switches, as there are also
a lot of them with dd if=/dev/zero of=/dev/null. As for cache issues,
yes, most of the time latencies are due to cache issues, but OMAP3 does
not have a VIVT cache. Also note that to have a precise idea of the
worst case latency with some load, you should let the test run a lot of
time, sometimes the worst case takes time to happen.


-- 
                                                                Gilles.