Re: [RFC 0/4] dynamically allocate arch specific system vectors

[Ingo Molnar <mingo@elte.hu>]

* Eric W. Biederman <ebiederm@xmission.com> wrote:

> Jack Steiner <steiner@sgi.com> writes:
> 
> > On Wed, Sep 17, 2008 at 03:15:07PM -0700, Eric W. Biederman wrote:
> >> Jack Steiner <steiner@sgi.com> writes:
> >> 
> >> > On Wed, Sep 17, 2008 at 12:15:42PM -0700, H. Peter Anvin wrote:
> >> >> Dean Nelson wrote:
> >> >> >
> >> >> >    sgi-gru driver
> >> >> >
> >> >> >The GRU is not an actual external device that is connected to an IOAPIC.
> >> >> >The gru is a hardware mechanism that is embedded in the node controller
> >> >> >(UV hub) that directly connects to the cpu socket. Any cpu (with 
> >> >> >permission)
> >> >> >can do direct loads and stores to the gru. Some of these stores will
> > result
> >> >> >in an interrupt being sent back to the cpu that did the store.
> >> >> >
> >> >> >The interrupt vector used for this interrupt is not in an IOAPIC. Instead
> >> >> >it must be loaded into the GRU at boot or driver initialization time.
> >> >> >
> >> >> 
> >> >> Could you clarify there: is this one vector number per CPU, or are you 
> >> >> issuing a specific vector number and just varying the CPU number?
> >> >
> >> > It is one vector for each cpu.
> >> >
> >> > It is more efficient for software if the vector # is the same for all cpus
> >> Why?  Especially in terms of irq counting that would seem to lead to cache
> >> line conflicts.
> >
> > Functionally, it does not matter. However, if the IRQ is not a per-cpu IRQ, a
> > very large number of IRQs (and vectors) may be needed. The GRU requires 32
> > interrupt
> > lines on each blade. A large system can currently support up to 512 blades.
> 
> Every vendor of high end hardware is saying they intend to provide
> 1 or 2 queues per cpu and 1 irq per queue.  So the GRU is not special in
> that regard.  Also a very large number of IRQs is not a problem as
> soon as we start dynamically allocating them, which is currently
> in progress.
> 
> Once we start dynamically allocating irq_desc structures we can put
> them in node-local memory and guarantee there is no data shared between
> cpus.
> 
> > After looking thru the MSI code, we are starting to believe that we should
> > separate
> > the GRU requirements from the XPC requirements. It looks like XPC can easily use
> > the MSI infrastructure.  XPC needs a small number of IRQs, and interrupts are
> > typically
> > targeted to a single cpu. They can also be retargeted using the standard
> > methods.
> 
> Alright. 
> 
> I would be completely happy if there were interrupts who's affinity we can
> not change, and are always targeted at a single cpu.
> 
> > The GRU, OTOH, is more like a timer interrupt or like a co-processor interrupt.
> > GRU interrupts can occur on any cpu using the GRU. When interrupts do occur, all
> > that
> > needs to happen is to call an interrupt handler. I'm thinking of something like
> > the following:
> >
> > 	- permanently reserve 2 system vectors in include/asm-x86/irq_vectors.h
> > 	- in uv_system_init(), call alloc_intr_gate() to route the
> > 	  interrupts to a function in the file containing uv_system_init().
> > 	- initialize the GRU chipset with the vector, etc, ...
> > 	- if an interrupt occurs and the GRU driver is NOT loaded, print
> > 	  an error message (rate limited or one time)
> >
> > 	- provide a special UV hook for the GRU driver to register/deregister a
> > 	  special callback function for GRU interrupts
> 
> That would work.  So far the GRU doesn't sound that special.
> 
> For a lot of this I would much rather solve the general case on this 
> giving us a solution that works for all high end interrupts rather 
> than one specific solution just for the GRU.  Especially since it 
> looks like we have most of the infrastructure already present to solve 
> the general case and we have to develop and review the specific case 
> from scratch.

ok, great.

Dean, just to make sure the useful bits are not lost now that the 
direction has been changed: could you please repost the patchset but 
without the driver API bits? It's still all a nice and useful 
generalization and cleanup of the x86 vector allocation code, and we can 
check it in -tip how well it works in practice.

	Ingo