[Xenomai-help] Interrupt latency questions

[Xenomai-help] Interrupt latency questions

[Antonio Del Cinque]

[-- Attachment #1: Type: text/plain, Size: 2637 bytes --]

Hi all,
I need confirmations on the ipipe model, to interpret test results. Excuse
me, I'm new to this
kind of work, I only read introductory articles on the adeos/xenomai
architecture.

I'm testing Xenomai on an e300 Core PowerPC, and, to get lowest latencies, I
realized an RTDM
driver which handles an external timer interrupt:

-sh-2.05b# cat /proc/ipipe/Xenomai
       +----- Handling ([A]ccepted, [G]rabbed, [W]ired, [D]iscarded)
       |+---- Sticky
       ||+--- Locked
       |||+-- Exclusive
       ||||+- Virtual
[IRQ]  |||||
  75:  W..X.
 512:  W..XV
 515:  W...V

IRQ 75 is the timer interrupt. It is not handled in the Linux domain:

-sh-2.05b# cat /proc/ipipe/Linux | grep " 75"
  75:  A....

The Xenomai interrupt handler simply toggles a pin
generating a squarewave measured by an oscilloscope, so this is a jitter
test:

static int demo_interrupt(rtdm_irq_t *irq_context)
{
    int ret = RTDM_IRQ_HANDLED;
    struct gpt_drv_context *ctx;

    ctx = rtdm_irq_get_arg(irq_context, struct gpt_drv_context);

    // re-enable timer interrupt
    out_be32((ctx->gpt) + GPT_STATUS, GPT_STATUS_TEXP);

    // toggle GPIO
    if (read_pin(PIN, ctx)) {
        set_pin_low(PIN, ctx);
    } else {
        set_pin_high(PIN, ctx);
    }

    return ret;
}

The test shows a 2/3 uS jitter with no load.

Then I open an ssh connection with the target, and run a "cat /dev/zero >
file". The target
is mounted on an NFS root filesystem, so this generates a high number of
fast ethernet interrupts. Now the oscilloscope shows 15 uS jitter in the
worst case,
and a visible variance. I would assume this effect as increased max latency.

The ethernet interrupt is handled only in Linux domain by the original
vanilla handler,
but Linux should be stalled/preempted when the timer interrupt occurs. So I
assume
that latency is to be justified by the time it takes to adeos to record an
ethernet interrupt event in the i-logs when the timer interrupt is executing
or ready to
execute. Is this correct?

Assuming it is correct, what i-logs are concerned in the case of the
ethernet interrupt?
Is only the Xenomai i-log, only the Linux i-log, or both i-logs concerned? I
removed the
Interrupt Shield, but did not notice jitter differences.

Is there any way to lower max latency for the timer interrupt?

May I assume the max jitter measured is the upper bound, when there is only
this Xenomai
interrupt handler, and no other real time tasks?

May I assume that increasing the number of non real time related interrupts
in the system
would determine an increased mean latency, but not an increased max latency?

Thank you very much,
Antonio

[-- Attachment #2: Type: text/html, Size: 3346 bytes --]

Re: [Xenomai-help] Interrupt latency questions

[Fillod Stephane]

Antonio Del Cinque wrote:
[...]
>Then I open an ssh connection with the target, and run a "cat /dev/zero
> file". The target 
>is mounted on an NFS root filesystem, so this generates a high number
of 
>fast ethernet interrupts. Now the oscilloscope shows 15 uS jitter in
the worst case, 
>and a visible variance. I would assume this effect as increased max
latency.

This figure has already been seen with SoC in the range of the e300, 
and sometime worse.

>The ethernet interrupt is handled only in Linux domain by the original
vanilla handler, 
>but Linux should be stalled/preempted when the timer interrupt occurs.
So I assume 
>that latency is to be justified by the time it takes to adeos to record
an 
>ethernet interrupt event in the i-logs when the timer interrupt is
executing or ready to
>execute. Is this correct? 

Cache miss and TLB miss are also good latency killers.
To some extent, long DMA transfers can bog down the system bus.

>Is there any way to lower max latency for the timer interrupt?

Has anybody tried to play with cache locking?

-- 
Stephane

Re: [Xenomai-help] Interrupt latency questions

[Antonio Del Cinque]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]

Thank you,
I think your considerations are the right ones.

The DMA controller is not generating any interrupt, so I assume it is not
actually working. I'm trying to verify that the jitter is caused by
cache/TLB misses. I don't have much reference on how to lock the cache, so
I'll just try to have the cache dirty in any other way to see if it causes
the same jitter.

Antonio

On Mon, Sep 22, 2008 at 11:15 AM, Fillod Stephane <
stephane.fillod@domain.hid> wrote:

> Antonio Del Cinque wrote:
> [...]
> >Then I open an ssh connection with the target, and run a "cat /dev/zero
> > file". The target
> >is mounted on an NFS root filesystem, so this generates a high number
> of
> >fast ethernet interrupts. Now the oscilloscope shows 15 uS jitter in
> the worst case,
> >and a visible variance. I would assume this effect as increased max
> latency.
>
> This figure has already been seen with SoC in the range of the e300,
> and sometime worse.
>
> >The ethernet interrupt is handled only in Linux domain by the original
> vanilla handler,
> >but Linux should be stalled/preempted when the timer interrupt occurs.
> So I assume
> >that latency is to be justified by the time it takes to adeos to record
> an
> >ethernet interrupt event in the i-logs when the timer interrupt is
> executing or ready to
> >execute. Is this correct?
>
> Cache miss and TLB miss are also good latency killers.
> To some extent, long DMA transfers can bog down the system bus.
>
> >Is there any way to lower max latency for the timer interrupt?
>
> Has anybody tried to play with cache locking?
>
> --
> Stephane
>

[-- Attachment #2: Type: text/html, Size: 2126 bytes --]

Re: [Xenomai-help] Interrupt latency questions

[Gilles Chanteperdrix]

Antonio Del Cinque wrote:
> Thank you,
> I think your considerations are the right ones.
> 
> The DMA controller is not generating any interrupt, so I assume it is not
> actually working. I'm trying to verify that the jitter is caused by
> cache/TLB misses. I don't have much reference on how to lock the cache, so
> I'll just try to have the cache dirty in any other way to see if it causes
> the same jitter.

A good way to do this is to use the cache calibrator:
http://homepages.cwi.nl/~manegold/Calibrator/

-- 
                                                 Gilles.

Re: [Xenomai-help] Interrupt latency questions

[Antonio Del Cinque]

[-- Attachment #1: Type: text/plain, Size: 3329 bytes --]

Hi,
sorry, I needed time to implement your suggestions, but finally I got some
results. I checked for cache related issues, and found that latency is
highly related to cache misses.

I found the way to disable the instruction cache (the e300 powerpc core has
a 32 KB i-cache and a 32 KB d-cache, but I couldn't disable the d-cache
because of segmentation faults), and I changed the test to measure interrupt
latency.

Now I have a Xenomai real time task. The task is periodic (I used
rt_task_set_periodic() and rt_task_wait_period()) and, at every period, it
toggles a GPIO pin configured as an output, by calling the GPIO rtdm driver
ioctl command. The pin is short circuited with another pin which is
configured as an input, generating an interrupt in the powerpc. The
interrupt routine, registerd via rtdm_irq_request() in the same GPIO rtdm
driver, toggles another GPIO pin configured as an output, so I see the two
IO pins on the oscilloscope, the first generating the interrupt, and the
second indicating that the interrupt routine has been executed.

When the cache is enabled I see a latency between 2 and 5 uS with no load,
but if I create a non real time load the latency raises (for example to 12
uS with the cache calibrator running), and this is non deterministic, since
it dependes on the cache use.

When I disable the i-cache the latency goes to 16 uS with no load, this is
11 to 14 uS over latency! Non real time task now only sligthly affect the
latency, but I have the following problem: if I start real time tasks (for
example I run /usr/local/xenomai/share/xenomai/testsuite/cyclic/run) the
GPIO interrupt latency is not commonly affected, but occasionally it gets to
50 uS. I can't explain this. Could you help me?

The delay is not continuos, it jumps from 16-20 to 38-50 uS. Now the i-cache
is disabled, so it seems another interrupt routine, running in Xenomai
domain, gets interleaved with the GPIO interrupt rountine.

Is there any way to track these kind of latency in the ipipe, a kind of
worst latency case tracker? I see only "virtual" IRQs 512 and 515 running in
Xenomai domain. I cannot see where they are set up. I guess one of them is
the system timer, and I know this is commonly driven by the powerpc
decrementer. Is the GPIO interrupt latency due to the system timer
interrupt? If this is the case, where can I disable the system timer in
Xenomai, or modify the interrupt routine? I must run in interrupt context
because of latency requirements (they asked me for deterministic latencies
under 20 uS).

Thank you, very much
Antonio





On Tue, Sep 23, 2008 at 10:25 AM, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> Antonio Del Cinque wrote:
> > Thank you,
> > I think your considerations are the right ones.
> >
> > The DMA controller is not generating any interrupt, so I assume it is not
> > actually working. I'm trying to verify that the jitter is caused by
> > cache/TLB misses. I don't have much reference on how to lock the cache,
> so
> > I'll just try to have the cache dirty in any other way to see if it
> causes
> > the same jitter.
>
> A good way to do this is to use the cache calibrator:
> http://homepages.cwi.nl/~manegold/Calibrator/<http://homepages.cwi.nl/%7Emanegold/Calibrator/>
>
> --
>                                                  Gilles.
>

[-- Attachment #2: Type: text/html, Size: 3969 bytes --]

Re: [Xenomai-help] Interrupt latency questions

[Gilles Chanteperdrix]

Antonio Del Cinque wrote:
> Is there any way to track these kind of latency in the ipipe, a kind of
> worst latency case tracker? I see only "virtual" IRQs 512 and 515 running in
> Xenomai domain. I cannot see where they are set up. I guess one of them is
> the system timer, and I know this is commonly driven by the powerpc
> decrementer. Is the GPIO interrupt latency due to the system timer
> interrupt? If this is the case, where can I disable the system timer in
> Xenomai, or modify the interrupt routine? I must run in interrupt context
> because of latency requirements (they asked me for deterministic latencies
> under 20 uS).

Yes, there is the I-pipe tracer:
http://www.xenomai.org/index.php/I-pipe:Tracer

Xenomai latency test has a -f option which triggers a trace for the
worst case.

-- 
                                                 Gilles.

Re: [Xenomai-help] Interrupt latency questions

[Wolfgang Grandegger]

Hello,

Antonio Del Cinque wrote:
> Hi,
> sorry, I needed time to implement your suggestions, but finally I got
> some results. I checked for cache related issues, and found that latency
> is highly related to cache misses.
> 
> I found the way to disable the instruction cache (the e300 powerpc core
> has a 32 KB i-cache and a 32 KB d-cache, but I couldn't disable the
> d-cache because of segmentation faults), and I changed the test to
> measure interrupt latency.

What system are you exactly using. Your latency figures are already
really good.

> Now I have a Xenomai real time task. The task is periodic (I used
> rt_task_set_periodic() and rt_task_wait_period()) and, at every period,
> it toggles a GPIO pin configured as an output, by calling the GPIO rtdm
> driver ioctl command. The pin is short circuited with another pin which
> is configured as an input, generating an interrupt in the powerpc. The
> interrupt routine, registerd via rtdm_irq_request() in the same GPIO
> rtdm driver, toggles another GPIO pin configured as an output, so I see
> the two IO pins on the oscilloscope, the first generating the interrupt,
> and the second indicating that the interrupt routine has been executed.

I have done similar tests with gpioirqbench:

  http://www.denx.de/wiki/DULG/AN2008_03_Xenomai_gpioirqbench

> When the cache is enabled I see a latency between 2 and 5 uS with no
> load, but if I create a non real time load the latency raises (for
> example to 12 uS with the cache calibrator running), and this is non
> deterministic, since it dependes on the cache use.

Additionally try to run "while ls; do ls /bin; done" in a telnet session.

> When I disable the i-cache the latency goes to 16 uS with no load, this
> is 11 to 14 uS over latency! Non real time task now only sligthly affect
> the latency, but I have the following problem: if I start real time
> tasks (for example I run
> /usr/local/xenomai/share/xenomai/testsuite/cyclic/run) the GPIO
> interrupt latency is not commonly affected, but occasionally it gets to
> 50 uS. I can't explain this. Could you help me?

Running the systems with caches disabled does make little sense because
it would make the system very very slow (apart from the fact that it's
not easily possible with Linux).

> The delay is not continuos, it jumps from 16-20 to 38-50 uS. Now the
> i-cache is disabled, so it seems another interrupt routine, running in
> Xenomai domain, gets interleaved with the GPIO interrupt rountine.
> 
> Is there any way to track these kind of latency in the ipipe, a kind of
> worst latency case tracker? I see only "virtual" IRQs 512 and 515
> running in Xenomai domain. I cannot see where they are set up. I guess
> one of them is the system timer, and I know this is commonly driven by
> the powerpc decrementer. Is the GPIO interrupt latency due to the system
> timer interrupt? If this is the case, where can I disable the system
> timer in Xenomai, or modify the interrupt routine? I must run in
> interrupt context because of latency requirements (they asked me for
> deterministic latencies under 20 uS).

As I see it, the main latencies are caused by cache refills. If the
latencies are not good enough, choose a faster system (more MHz, bigger
L1 or L2 cache or faster memory) but 20 us is very challenging already
for a PC. Moving your task to kernel space or even to the ISR gives
better result, of course, as you can see from:

  http://www.denx.de/wiki/DULG/AN2008_03_Xenomai_gpioirqbench

Wolfgang.