Interrupt Latencies

Interrupt Latencies

[Schaefer Dr, Frank-Rene ()]

Hello. 

Having read "Moving interrupts to threads" at

           http://lwn.net/Articles/302043/

I expected to reduce interrupt latency during a SPI 
communication by handling the transmit-receive in a 
'quick_check_handler' using 

           request_threaded_irq(...);


However, the difference in latency to "request_irq(...)" 
is not measurable. It is still in the range from 60 to 110us
on a 1.6 GHz Atom CPU. From "linux/interrupt.h" I conclude 
that depending on 

           CONFIG_GENERIC_HARDIQS

either 'request_irq' is mapped to 'request_threaded_irq' 
or vice versa. We are able to measure the latency precisely
as the difference of the time when the interrupt pin 'IN'
is raised and we raise our response pin 'OUT' as shown below.

      pin IN        .-------------------------
      ______________|
                    
      pin OUT                     .-----------
      ____________________________|
                          
                    |<- latency ->|

Could anyone point to locations in the kernel so that I can
precisely understand the mechanisms that cause the latency? It 
is totally incomprehensible to me why the 'quick_check_handler' 
must have a latency of 60us at min. (that are many thousand 
instructions).

Any help and comments are greatly appreciated.

Best Regards

Frank

Re: Interrupt Latencies

[Clemens Ladisch]

Schaefer Dr, Frank-Rene wrote:
> Having read "Moving interrupts to threads" at
> 
>            http://lwn.net/Articles/302043/
> 
> I expected to reduce interrupt latency during a SPI
> communication by handling the transmit-receive in a
> 'quick_check_handler' using
> 
>            request_threaded_irq(...);
> 
> However, the difference in latency to "request_irq(...)"
> is not measurable.

Threaded interrupts reduce the latency for _other_ interrupts because
most of the code has been moved into the thread.  Without threaded
interrupts, the same could would be in the actual interrupt handler
(the equivalent of the quick check handler).

> It is totally incomprehensible to me why the 'quick_check_handler'
> must have a latency of 60us at min. (that are many thousand
> instructions).

The sytem's interrupt handling and the I/O accesses aren't fast.
Furthermore, waking the CPU up, or changing its frequency, can be quite
slow (but I don't know how much in the case of your Atom).  You could
try reducing the interrupt latency by not letting the CPU get idle but
executing some low-priority busy loop.


Regards,
Clemens

RE: Interrupt Latencies

[Schaefer Dr, Frank-Rene ()]

In response to Thomas Gleixner:

> Interrupt latency depends on various factors:
> 
>   - Interrupt disabled code regions
Could you tell us how this could be detected or measured?
Is there a central place, where we could toggle a pin to 
see when interrupts are enabled/disabled? 

>   - Concurrent interrupts and the ordering of handling
We are only considering the best times that we measure.
The only other interrupts are SATA and system timer, I guess.

>   - Deep idle states
You mean 'suspend' or some type of CPU sleep. Could you elaborate?
We do not suspend. This should also only be the case for the 
first chunk that is transmitted. But the latencies remain
over longer time spans.

>   - Bus contention
Nothing else is running. 

>   - Cache misses
We cannot make any statements about that. But, we are **not** 
working on huge data or code regions.

> >
> >       pin IN        .-------------------------
> >       ______________|
> >
> >       pin OUT                     .-----------
> >       ____________________________|
> >
> >                     |<- latency ->|
> >

> How is that interrupt connected to the CPU/chipset? 
The port is part of the chipset connected internally 
via PCI.

> Which driver(s) is/are involved ? 
http://lxr.free-electrons.com/source/drivers/gpio/pl061.c

> How is the pin OUT accessed from the driver?
gpio_set_value(Pin, Value);

we measured the delay and could find that it was in the range
of some nano seconds.

Best Regards

Frank Schäfer and Joachim Becker.

RE: Interrupt Latencies

[Thomas Gleixner]

Frank,

On Thu, 24 Feb 2011, Schaefer Dr, Frank-Rene () wrote:

> In response to Thomas Gleixner:
> 
> > Interrupt latency depends on various factors:
> > 
> >   - Interrupt disabled code regions
> Could you tell us how this could be detected or measured?
> Is there a central place, where we could toggle a pin to 
> see when interrupts are enabled/disabled? 

ftrace allows you to analyse that.
 
> >   - Concurrent interrupts and the ordering of handling
> We are only considering the best times that we measure.

Ok.

> >   - Deep idle states
> You mean 'suspend' or some type of CPU sleep. Could you elaborate?
> We do not suspend. This should also only be the case for the 
> first chunk that is transmitted. But the latencies remain
> over longer time spans.

NOHZ idle can push the cpu into deeper power states (C2/C3). But there
are also BIOSes which do agressive power management behind the kernels
back.

Turn off CONFIG_NOHZ and add "processor.max_cstate=1" to the kernel
command line. You might also try "idle=poll" for a test.
 
> >   - Bus contention
> Nothing else is running. 

That does not guarantee anything. Think DMA.

> > How is that interrupt connected to the CPU/chipset? 
> The port is part of the chipset connected internally 
> via PCI.
> 
> > Which driver(s) is/are involved ? 
> http://lxr.free-electrons.com/source/drivers/gpio/pl061.c
> 
> > How is the pin OUT accessed from the driver?
> gpio_set_value(Pin, Value);
> 
> we measured the delay and could find that it was in the range
> of some nano seconds.

Is the SPI controller part of the chipset or comes the interrupt via
one of the GPIOs as well?

Thanks,

	tglx

Re: Interrupt Latencies

[Thomas Gleixner]

On Tue, 22 Feb 2011, Schaefer Dr, Frank-Rene () wrote:
> Having read "Moving interrupts to threads" at
> 
>            http://lwn.net/Articles/302043/
> 
> I expected to reduce interrupt latency during a SPI 
> communication by handling the transmit-receive in a 
> 'quick_check_handler' using 
> 
>            request_threaded_irq(...);

The quick check handler has the same latencies as the normal handler
of an interrupt requested by request_irq.
 
Interrupt latency depends on various factors:

  - Interrupt disabled code regions
  - Concurrent interrupts and the ordering of handling
  - Deep idle states
  - Bus contention
  - Cache misses

The maximum latency is the worst case of all the above added together.

> or vice versa. We are able to measure the latency precisely
> as the difference of the time when the interrupt pin 'IN'
> is raised and we raise our response pin 'OUT' as shown below.
> 
>       pin IN        .-------------------------
>       ______________|
>                     
>       pin OUT                     .-----------
>       ____________________________|
>                           
>                     |<- latency ->|
> 
> Could anyone point to locations in the kernel so that I can
> precisely understand the mechanisms that cause the latency? It 

There is no single mechanism.

> is totally incomprehensible to me why the 'quick_check_handler' 
> must have a latency of 60us at min. (that are many thousand 
> instructions).

How is that interrupt connected to the CPU/chipset? Which driver(s)
is/are involved ? How is the pin OUT accessed from the driver?

Thanks,

	tglx

RE: Interrupt Latencies

[Schaefer Dr, Frank-Rene ()]

We are using GPIO pins and map them to IRQ. 
The underlying driver seems to use message signaled
interrupts. 


-----Original Message-----
From: Thomas Gleixner [mailto:tglx@linutronix.de] 
Sent: Tuesday, February 22, 2011 3:39 PM
To: Schaefer Dr, Frank-Rene ()
Cc: linux-kernel@vger.kernel.org
Subject: Re: Interrupt Latencies

On Tue, 22 Feb 2011, Schaefer Dr, Frank-Rene () wrote:
> Having read "Moving interrupts to threads" at
> 
>            http://lwn.net/Articles/302043/
> 
> I expected to reduce interrupt latency during a SPI 
> communication by handling the transmit-receive in a 
> 'quick_check_handler' using 
> 
>            request_threaded_irq(...);

The quick check handler has the same latencies as the normal handler
of an interrupt requested by request_irq.
 
Interrupt latency depends on various factors:

  - Interrupt disabled code regions
  - Concurrent interrupts and the ordering of handling
  - Deep idle states
  - Bus contention
  - Cache misses

The maximum latency is the worst case of all the above added together.

> or vice versa. We are able to measure the latency precisely
> as the difference of the time when the interrupt pin 'IN'
> is raised and we raise our response pin 'OUT' as shown below.
> 
>       pin IN        .-------------------------
>       ______________|
>                     
>       pin OUT                     .-----------
>       ____________________________|
>                           
>                     |<- latency ->|
> 
> Could anyone point to locations in the kernel so that I can
> precisely understand the mechanisms that cause the latency? It 

There is no single mechanism.

> is totally incomprehensible to me why the 'quick_check_handler' 
> must have a latency of 60us at min. (that are many thousand 
> instructions).

How is that interrupt connected to the CPU/chipset? Which driver(s)
is/are involved ? How is the pin OUT accessed from the driver?

Thanks,

	tglx

RE: Interrupt Latencies

[Thomas Gleixner]

On Tue, 22 Feb 2011, Schaefer Dr, Frank-Rene () wrote:

Please do not top post.

> We are using GPIO pins and map them to IRQ. 

How are the GPIO pins accessed ? Are they memory mapped registers or what ?

> The underlying driver seems to use message signaled interrupts.

That does not answer which driver is involved. W/o seing the code I
can't tell much.

Thanks,

	tglx