Lapic vector priorites

Lapic vector priorites

[john stultz]

Hey Thomas,
	I got a question this weekend about irq priorities where it was looking
like the hrtimer_interrupt was bring delayed under heavy load spikes.
The issue hasn't been conclusively determined, but this reminded me of
the old issue that cropped up occasionally with 2.4 (maybe with
clustered apic addressing?) where we got lost-ticks that were caused by
irq storms, due to the timer interrupt being lower priority then the
scsi controller.

I'm really not that familiar with the lapic code, but reading over some
of the documentation I could find through searches, and then looking at
the lapic vector layout in the kernel, I'm now a little curious.

>From what I've read the irq priority is set by the vector, where 0x0 is
the highest and 0xFF is the lowest.

In irq_vectors.h I see:
#define NMI_VECTOR			0x02
#define MCE_VECTOR			0x12

So really critical stuff being very low. Makes sense. Then:

#define FIRST_EXTERNAL_VECTOR		0x20

Then a gap. Then:

#define IRQ0_VECTOR		((FIRST_EXTERNAL_VECTOR + 16) & ~15)
...
#define IRQ15_VECTOR			(IRQ0_VECTOR + 15)

Ok, so IRQ0 is at vector 0x30 and IRQ15 is at 0x3f

Then there's other cpu specific bits way up at 0xf3-0xff.

And then I see:
/*
 * Local APIC timer IRQ vector is on a different priority level,
 * to work around the 'lost local interrupt if more than 2 IRQ
 * sources per level' errata.
 */
#define LOCAL_TIMER_VECTOR		0xef


So, from this it seems that the lapic timer is going to be a lower
priority then most of the irq space (or atleast irqs 0-15) . Making it
possible to see timer latencies from irq storms.  Is that right?  Or do
all the irqs really come in via:
#define IRQ_WORK_VECTOR			0xf6


I realize from the comment that the lapic needs to be on its own range
to avoid the errata, but could that range be 0x2X (moving whatever goes
in the 0x2X range elsewhere)?  I did a brief test setting the lapic to
0x2f and my simple test machine booted, so it seems this might be
possible.

I realize from all the bug reports over the last number of years how
fragile the lapic code is (or more accurately how poor and quirky the
hardware is), but it seemed like having timer irqs be behind other
devices would be a possible latency source that would likely have -rt
effects (I looked, but I don't see anything in the -rt patch set playing
with irq vectors - maybe this issue is avoided in other ways?).

Am I just misunderstanding all of this?

thanks
-john

Re: Lapic vector priorites

[Maciej W. Rozycki]

On Sun, 22 Jan 2012, john stultz wrote:

> I'm really not that familiar with the lapic code, but reading over some
> of the documentation I could find through searches, and then looking at
> the lapic vector layout in the kernel, I'm now a little curious.
> 
> >From what I've read the irq priority is set by the vector, where 0x0 is
> the highest and 0xFF is the lowest.
> 
> In irq_vectors.h I see:
> #define NMI_VECTOR			0x02
> #define MCE_VECTOR			0x12

 Both these vectors are hardwired internally in the CPU and irrelevant to 
the APIC interrupt priority levels as these apply to ordinary interrupts 
only.  For these the higher the vector value, the higher the priority (I 
won't get into the gory details, my memory has already faded about that 
and I'd have to resort to documentation to avoid confusion).

 The MCE is actually a processor exception and as such does not go through 
the APIC at all.  It's either signalled internally by the CPU (to record 
an internal malfunction) or via an external signal (to raise a bus error 
or a similar event).

 The NMI is the non-maskable interrupt and is driven by the local APIC on 
a separate processor's internal line (back in the 1990s where discrete 
local APICs were used this would actually be a separate board signal to 
the CPU) and it's the CPU that gives this line the priority above an 
ordinary interrupt that may be asserted at the same time.  For an NMI the 
CPU does not drive an INTA cycle (again, internally for modern processors) 
on its bus which the local APIC would respond with an interrupt vector to.

 I hope this helps.

  Maciej