Latency measurement with oscilloscope, randomly high latency times

Latency measurement with oscilloscope, randomly high latency times

[Camera.Geomatics]

Hi

We try to achieve hard real time requirements with a 
FPGA->IRQ->ISR->User-Space->Kernel-Module->FPGA loop.

Kernel IRQ handler which toggles IO pins is executed by a Hardware 
interrupt every 200us (FPGA). Priority is set to 99 via chrt ?f ?p99 
pid-module.

Problem:
The test is working (Pins are toggled in time, <50us), but for some reason 

we have randomly (every 2-3sec) big latency times of several ms.

We sometimes get already interrupted in our IRQ handler (which we 
registered with request_irq) before we wakeup our user space task. 

Kernel Version: linux-socfpga 3.10.37-ltsi-rt37
Development board: Altera Cyclone V SoC Development Kit

Do we have to somehow set the priority 99 for the IRQ handler seperately? 
Is there any kind of priority inheritance from user space task to kernel 
module?

Best Regards
Hannes

Re: Latency measurement with oscilloscope, randomly high latency times

[Frank Rowand]

On 12/18/2014 7:12 AM, Camera.Geomatics@leica-geosystems.com wrote:
> Hi
> 
> We try to achieve hard real time requirements with a 
> FPGA->IRQ->ISR->User-Space->Kernel-Module->FPGA loop.
> 
> Kernel IRQ handler which toggles IO pins is executed by a Hardware 
> interrupt every 200us (FPGA). Priority is set to 99 via chrt ?f ?p99 
> pid-module.
> 
> Problem:
> The test is working (Pins are toggled in time, <50us), but for some reason 
> 
> we have randomly (every 2-3sec) big latency times of several ms.

Often, the easiest way to identify the big latencies is to run
cyclictest, using its debug facilities such as ftrace triggering.

For a several ms latency you might want to start with hwlatdetect
to check for SMIs.  But it is possible to get large latencies
from other causes than SMIs.

But I would start with adjusting the IRQ handler priority, as you
suspected you might need to (see below).

> 
> We sometimes get already interrupted in our IRQ handler (which we 
> registered with request_irq) before we wakeup our user space task. 
> 
> Kernel Version: linux-socfpga 3.10.37-ltsi-rt37
> Development board: Altera Cyclone V SoC Development Kit
> 
> Do we have to somehow set the priority 99 for the IRQ handler seperately? 
> Is there any kind of priority inheritance from user space task to kernel 
> module?

First off, you should not use priority 99 unless you _really_ understand
what that is doing to the other kernel threads that are priority 99 (they
have that priority for a reason).

Yes, you need to set the priority of your IRQ handler thread to the
value that you decide is appropriate. If your IRQ handler or the
other parts of your real time application depend on services from
other kernel threads, you may need to adjust the priority of those
threads.

"The user space task" is causing the kernel module to execute by doing
a system call, correct?  Thus the current task while executing the
kernel module code is the "user space task", so the priority is the
"user space task" priority.  This is assuming that there is nothing
in the path from the syscall to the kernel module acting on the FPGA
that is asynchronous (for example, the kernel module could pass a
message to a kernel thread that acts on the FPGA).

> 
> Best Regards
> Hannes
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>