about interrupt latency

about interrupt latency

[Francesco Oppedisano]

Hi,
i'm trying to estimate the interrupt latency (time between hardware
interrrupt and the start of the ISR) of a linux kernel 2.4.29 and i
used a simple tecnique: inside the do_timer_interrupt i read the 8259
counter to obtain the elapsed time.
By this mean i found a latency of about 6/7 microseconds that is very
similar to the time measured in some articles but with CPU much slower
while i expected the latency was shorter on faster CPUs.
So, my questions are: 
1)what's the depency between the interrupt latency and the CPU speed?
2)what are the factors at the origin of th interrupt latency?

Than u very much

Francesco Oppedisano

Re: about interrupt latency

[linux-os]

On Tue, 8 Mar 2005, Francesco Oppedisano wrote:

> Hi,
> i'm trying to estimate the interrupt latency (time between hardware
> interrrupt and the start of the ISR) of a linux kernel 2.4.29 and i
> used a simple tecnique: inside the do_timer_interrupt i read the 8259
> counter to obtain the elapsed time.
> By this mean i found a latency of about 6/7 microseconds that is very
> similar to the time measured in some articles but with CPU much slower
> while i expected the latency was shorter on faster CPUs.
> So, my questions are:
> 1)what's the depency between the interrupt latency and the CPU speed?
> 2)what are the factors at the origin of th interrupt latency?
>
> Than u very much
>
> Francesco Oppedisano

You can't measure interrupt latency that way even
though you may get the "correct" answer!

Make a simple module that uses IRQ7, the printer-port
interrupt. Inside your ISR, you toggle one of the
printer-port bits. Program the printer port to
generate the interrupt when its control bit
is triggered.

Now, connect a function generator to toggle the
printer control bit. Also use this transition to
trigger an oscilloscope while looking at its trace
on one channel. Connect the other channel to the
bit that's being toggled in your ISR.

Observe the time between the trigger-trace and
the toggle-trace. That, minus the few nanoseconds
necessary to execute your ISR code, is the
interrupt latency when using that specific interrupt
source. PCI/Bus devices have lower latencies.

The CPU speed seems to have little to do with interrupt
latency now that we have fast CPUs. The limiting action
is the memory speed (front-side bus). You can seldom
count on having your ISR code inside the cache, so it
needs to be fetched. It also takes more cache-flushes
to switch from user-mode to a kernel stack, set up
new segments, etc. That's the reason why you must
MEASURE the latency if it is important. Guessing that
an interrupt occurred when a timer went to zero, then
measuring the residual in that same ISR will give you
the wrong answers, altough in this case, it's probably
close.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.

Re: about interrupt latency

[Francesco Oppedisano]

On Tue, 8 Mar 2005 14:03:21 -0500 (EST), linux-os 
> You can't measure interrupt latency that way even
> though you may get the "correct" answer!

Why do you think the technique is not valid?

> Make a simple module that uses IRQ7, the printer-port
> interrupt. Inside your ISR, you toggle one of the
> printer-port bits. Program the printer port to
> generate the interrupt when its control bit
> is triggered.
> 
> Now, connect a function generator to toggle the
> printer control bit. Also use this transition to
> trigger an oscilloscope while looking at its trace
> on one channel. Connect the other channel to the
> bit that's being toggled in your ISR.
> 
> Observe the time between the trigger-trace and
> the toggle-trace. That, minus the few nanoseconds
> necessary to execute your ISR code, is the
> interrupt latency when using that specific interrupt
> source. PCI/Bus devices have lower latencies.

I do not have such an instrumentation :-(

> 
> The CPU speed seems to have little to do with interrupt
> latency now that we have fast CPUs. The limiting action
> is the memory speed (front-side bus). You can seldom
> count on having your ISR code inside the cache, so it
> needs to be fetched. It also takes more cache-flushes
> to switch from user-mode to a kernel stack, set up
> new segments, etc. That's the reason why you must
> MEASURE the latency if it is important. Guessing that
> an interrupt occurred when a timer went to zero, then
> measuring the residual in that same ISR will give you
> the wrong answers, altough in this case, it's probably
> close.
> 
Sorry but i cannot understand this:
i'm trying to estimate interrupt latency (i need only the order of
magnitude) by measuring the interrupt latency of the timer interrupt.
So when the 8254 counter reaches zero it issues an interrupt so i'm
sure that the time when the interrupt has been issued was when the
counter was to zero. From that time the counter returns to a value of
19832 (or something else) and at that time the countdown restarts.
When i reach the ISR i read the counter and obtain the
latency...what's wrong in this simple strategy?
Probably  i'm guessing  that the latency experienced by a timer
interrupt is the same as e.g the latency experienced by a NIC
interrupt, but remember that i need only a coarse measure.

Nevertheless, the information about the ISR in cache is very
interesting, probably it's what i was looking for....thank u.

Francesco Oppedisano

Re: about interrupt latency

[Zwane Mwaikambo]

Hi Francesco,

On Tue, 8 Mar 2005, Francesco Oppedisano wrote:

> i'm trying to estimate the interrupt latency (time between hardware 
> interrrupt and the start of the ISR) of a linux kernel 2.4.29 and i used 
> a simple tecnique: inside the do_timer_interrupt i read the 8259 counter 
> to obtain the elapsed time. By this mean i found a latency of about 6/7 
> microseconds that is very similar to the time measured in some articles 
> but with CPU much slower while i expected the latency was shorter on 
> faster CPUs. So, my questions are: 1)what's the depency between the 
> interrupt latency and the CPU speed? 2)what are the factors at the 
> origin of th interrupt latency?

At some cpu frequency point on i386 the main cause of your interrupt 
service latency will be in the interrupt controller and how long from irq 
assertion to the signal being recognised, resultant vector being 
dispatched to the processor and the necessary interrupt controller 
acknowledge steps required. This is also helped by the fact that the 
Linux/i386 interrupt vector stubs are very small and fast, so there isn't 
all that much code to execute to reach the ISR from the vector table. I'm 
not sure if you've tested this, but you may notice that timer interrupt 
via Local APIC will have lower dispatch latency than timer interrupt via 
i8259 only. But that's all at the lower end of the latency graph, you will 
most likely run into other sources on a busy system.

	Zwane

Re: about interrupt latency

[Francesco Oppedisano]

On Tue, 8 Mar 2005 12:09:58 -0700 (MST), Zwane Mwaikambo
<zwane@arm.linux.org.uk> wrote:

> At some cpu frequency point on i386 the main cause of your interrupt
> service latency will be in the interrupt controller and how long from irq
> assertion to the signal being recognised, resultant vector being
> dispatched to the processor and the necessary interrupt controller
> acknowledge steps required. This is also helped by the fact that the
> Linux/i386 interrupt vector stubs are very small and fast, so there isn't
> all that much code to execute to reach the ISR from the vector table. I'm
> not sure if you've tested this, but you may notice that timer interrupt
> via Local APIC will have lower dispatch latency than timer interrupt via
> i8259 only. But that's all at the lower end of the latency graph, you will
> most likely run into other sources on a busy system.
> 
>         Zwane
> 
Very interesting zwane....i haven't tested the local APIC....do you
think this dispatch time can vary with the system I/O load (many
pending interrupts in the PIC)?

I think the interrupt latency is influenced even by the code inside
the kernel: if a lot of code is running with interrupts disabled then
the interrupt latency will grow. Am i right?

So probably we can state that the factors influencing the interrupt latency are:
1)Dispatching time in the PIC 
2)Waiting time on the PIC (if there are pending interrupt of lower vector)
3)fetching ISR from main memory
4)wait time when CPU is running with disabled interrupt

Do U agree?

Thank u very much
Francesco Oppedisano

Re: about interrupt latency

[Zwane Mwaikambo]

On Wed, 9 Mar 2005, Francesco Oppedisano wrote:

> On Tue, 8 Mar 2005 12:09:58 -0700 (MST), Zwane Mwaikambo
> <zwane@arm.linux.org.uk> wrote:
> 
> > At some cpu frequency point on i386 the main cause of your interrupt
> > service latency will be in the interrupt controller and how long from irq
> > assertion to the signal being recognised, resultant vector being
> > dispatched to the processor and the necessary interrupt controller
> > acknowledge steps required. This is also helped by the fact that the
> > Linux/i386 interrupt vector stubs are very small and fast, so there isn't
> > all that much code to execute to reach the ISR from the vector table. I'm
> > not sure if you've tested this, but you may notice that timer interrupt
> > via Local APIC will have lower dispatch latency than timer interrupt via
> > i8259 only. But that's all at the lower end of the latency graph, you will
> > most likely run into other sources on a busy system.
> > 
> >         Zwane
> > 
> Very interesting zwane....i haven't tested the local APIC....do you
> think this dispatch time can vary with the system I/O load (many
> pending interrupts in the PIC)?

The PIC/i386 setup cannot really pend interrupts so i would say no i don't 
think the dispatch time would be affected.

> I think the interrupt latency is influenced even by the code inside
> the kernel: if a lot of code is running with interrupts disabled then
> the interrupt latency will grow. Am i right?

Yes that's correct, in fact this will be your major/main cause of 
interrupt latency.

> So probably we can state that the factors influencing the interrupt latency are:
> 1)Dispatching time in the PIC 
> 2)Waiting time on the PIC (if there are pending interrupt of lower vector)

With APIC/i386 this is more possible, if you're really interested you 
should try and calculate the dispatch time using the same system (must 
have an IOAPIC) and testing with the following combinations;

PIC only (noapic, nolapic)
PIC + LAPIC (noapic)
IOAPIC + LAPIC

You will probably find that the IOAPIC + LAPIC has the lowest dispatch 
time. Also worth noting that the LAPIC can queue upto 2 vectors (on P4), 
this allows for more interrupts to be ready for dispatch.

> 3)fetching ISR from main memory
> 4)wait time when CPU is running with disabled interrupt
> 
> Do U agree?

A bit more on (4)

4a) wait time when CPU is running firmware (e.g. SMI/SMM, VGA BIOSes etc)

Otherwise, yes i agree, good luck with your research.

	Zwane

Re: about interrupt latency

[zyphr]

On Sat, 12 Mar 2005 10:01:43 -0700 (MST), Zwane Mwaikambo
<zwane@arm.linux.org.uk> wrote:
> On Wed, 9 Mar 2005, Francesco Oppedisano wrote:
> 
> > On Tue, 8 Mar 2005 12:09:58 -0700 (MST), Zwane Mwaikambo
> > <zwane@arm.linux.org.uk> wrote:
> >
> > > At some cpu frequency point on i386 the main cause of your interrupt
> > > service latency will be in the interrupt controller and how long from irq
> > > assertion to the signal being recognised, resultant vector being
> > > dispatched to the processor and the necessary interrupt controller
> > > acknowledge steps required. This is also helped by the fact that the
> > > Linux/i386 interrupt vector stubs are very small and fast, so there isn't
> > > all that much code to execute to reach the ISR from the vector table. I'm
> > > not sure if you've tested this, but you may notice that timer interrupt
> > > via Local APIC will have lower dispatch latency than timer interrupt via
> > > i8259 only. But that's all at the lower end of the latency graph, you will
> > > most likely run into other sources on a busy system.
> > >
> > >         Zwane
> > >
> > Very interesting zwane....i haven't tested the local APIC....do you
> > think this dispatch time can vary with the system I/O load (many
> > pending interrupts in the PIC)?
> 
> The PIC/i386 setup cannot really pend interrupts so i would say no i don't
> think the dispatch time would be affected.
> 
> > I think the interrupt latency is influenced even by the code inside
> > the kernel: if a lot of code is running with interrupts disabled then
> > the interrupt latency will grow. Am i right?
> 
> Yes that's correct, in fact this will be your major/main cause of
> interrupt latency.
> 
> > So probably we can state that the factors influencing the interrupt latency are:
> > 1)Dispatching time in the PIC
> > 2)Waiting time on the PIC (if there are pending interrupt of lower vector)
> 
> With APIC/i386 this is more possible, if you're really interested you
> should try and calculate the dispatch time using the same system (must
> have an IOAPIC) and testing with the following combinations;
> 
> PIC only (noapic, nolapic)
> PIC + LAPIC (noapic)
> IOAPIC + LAPIC
> 
> You will probably find that the IOAPIC + LAPIC has the lowest dispatch
> time. Also worth noting that the LAPIC can queue upto 2 vectors (on P4),
> this allows for more interrupts to be ready for dispatch.
> 
> > 3)fetching ISR from main memory
> > 4)wait time when CPU is running with disabled interrupt
> >
> > Do U agree?
> 
> A bit more on (4)
> 
> 4a) wait time when CPU is running firmware (e.g. SMI/SMM, VGA BIOSes etc)
> 
> Otherwise, yes i agree, good luck with your research.
> 
>         Zwane
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

I am just a user, not sure if it is related.
But with CONFIG_X86_UP_APIC and/or  CONFIG_X86_UP_IOAPIC set my mouse
behaviour is totally different, when playing games everything feels
more "laggy".
And it gives me what gamers call "negative acceleration".