Excessive time to handle interrupts

Excessive time to handle interrupts

[Peter Chubb]

I'm seeing very long interrupt handling times here, on McKinley (HP
ZX2000) -- of the order of a few microseconds.

Profiling indicates more than 4000 cycles are being spent in
enable_irq() after each interrupt.  As enable_irq() disables interrupt
collection, it's a bit difficult to get profiling data within that
function, but I strongly suspect it's the readl() in unmask_irq()
that's taking the time -- and likewise in mask_irq().


Code like this in arch/ia64/kernel/iosapic.c

    spin_lock_irqsave(&iosapic_lock, flags);
    {
	writel(IOSAPIC_RTE_LOW(rte_index), addr + IOSAPIC_REG_SELECT);
        low32 = readl(addr + IOSAPIC_WINDOW);
        low32 &= ~(1 << IOSAPIC_MASK_SHIFT);    /* set only the mask bit */
	writel(low32, addr + IOSAPIC_WINDOW);
    }
    spin_unlock_irqrestore(&iosapic_lock, flags);

I don't know enough about the hardware to say for sure, is it
feasible to keep a soft copy of the register rather than do the read
all the time?  If that's not going to break the hardware, I'll code it
up and see if interrupt latencies go down.

--
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
The technical we do immediately,  the political takes *forever*

Re: Excessive time to handle interrupts

[David Mosberger]

>>>>> On Fri, 27 Feb 2004 10:44:25 +1100, Peter Chubb <peter@chubb.wattle.id.au> said:

  Peter> writel(IOSAPIC_RTE_LOW(rte_index), addr + IOSAPIC_REG_SELECT);
  Peter> low32 = readl(addr + IOSAPIC_WINDOW);
  Peter> low32 &= ~(1 << IOSAPIC_MASK_SHIFT);    /* set only the mask bit */
  Peter> writel(low32, addr + IOSAPIC_WINDOW);

  Peter> I don't know enough about the hardware to say for sure

See:

  http://www.intel.com/design/itanium/downloads/25135001.pdf

  Peter> is it feasible to keep a soft copy of the register rather
  Peter> than do the read all the time?  If that's not going to break
  Peter> the hardware, I'll code it up and see if interrupt latencies
  Peter> go down.

AFAIR, the I/O SAPIC interrupt masking is entirely under control of
the CPU (i.e., the hw never changes the mask of its own) so caching
sounds like a good idea.

FYI: I'm planning to do a q-syscollect enhancement which would let you
get at least a flat profile for code which disables interrupts (or
even interrupt collection).

	--david

RE: Excessive time to handle interrupts

[Chen, Kenneth W]

> FYI: I'm planning to do a q-syscollect enhancement which would
> let you get at least a flat profile for code which disables
> interrupts (or even interrupt collection).

Cool ...  I can really use this kind of tool (and seriously need
this kind of capability).  What can we do to expedite this?

- Ken

Re: Excessive time to handle interrupts

[Grant Grundler]

On Thu, Feb 26, 2004 at 04:06:07PM -0800, David Mosberger wrote:
> See:
> 
>   http://www.intel.com/design/itanium/downloads/25135001.pdf

I just reviewed section 2.3.2.2 I/O xAPIC Redirection Table.
(See Figure 2-4. I/O xAPIC Redirection Table Entry Definition)

...
> AFAIR, the I/O SAPIC interrupt masking is entirely under control of
> the CPU (i.e., the hw never changes the mask of its own) so caching
> sounds like a good idea.

ISTR the same thing and reviewing the section gave me the warm
fuzzies it is safe to do.

My only concern is other parts of the code which touch IO SAPIC
redirection table entries need to update the cached copy as well.
Here's a list of places that touch IOSAPIC_RTE_LOW:
	set_rte()
	mask_irq()
	unmask_irq()
	iosapic_set_affinity()


BTW, enable/disable_irq() are not barriers for IRQ delivery.
disable_irq() will return before IO SAPIC knows to mask IRQs.
mask/unmask_irq doesn't force the last writel out of the CPU
and adding a readl to flush the writel wouldn't help performance.

Avoiding these calls in the performance path is the best strategy.

grant