Re: [RFC 0/4] dynamically allocate arch specific system vectors

[Dimitri Sivanich <sivanich@sgi.com>]

On Tue, Sep 16, 2008 at 03:46:54PM -0500, Dean Nelson wrote:
> On Tue, Sep 16, 2008 at 10:24:48AM +0200, Ingo Molnar wrote:
> > 
> > * Dean Nelson <dcn@sgi.com> wrote:
> > 
> > > > while i understand the UV_BAU_MESSAGE case (TLB flushes are 
> > > > special), why does sgi-gru and sgi-xp need to go that deep? They are 
> > > > drivers, they should be able to make use of an ordinary irq just 
> > > > like the other 2000 drivers we have do.
> > > 
> > > The sgi-gru driver needs to be able to allocate a single irq/vector 
> > > pair for all CPUs even those that are not currently online. The sgi-xp 
> > > driver has similar but not as stringent needs.
> > 
> > why does it need to allocate a single irq/vector pair? Why is a regular 
> > interrupt not good?
> 
> When you speak of a 'regular interrupt' I assume you are referring to simply
> the irq number, with the knowledge of what vector and CPU(s) it is mapped to
> being hidden?
> 
> 
>     sgi-gru driver
> 
> The GRU is not an actual external device that is connected to an IOAPIC.
> The gru is a hardware mechanism that is embedded in the node controller
> (UV hub) that directly connects to the cpu socket. Any cpu (with permission)
> can do direct loads and stores to the gru. Some of these stores will result
> in an interrupt being sent back to the cpu that did the store.
> 
> The interrupt vector used for this interrupt is not in an IOAPIC. Instead
> it must be loaded into the GRU at boot or driver initialization time.
> 
> The OS needs to route these interrupts back to the GRU driver interrupt
> handler on the cpu that received the interrupt. Also, this is a performance
> critical path. There should be no globally shared cachelines involved in the
> routing.
> 
> The actual vector associated with the IRQ does not matter as long as it is
> a relatively high priority interrupt. The vector does need to be mapped to
> all of the possible CPUs in the partition. The GRU driver needs to know
> vector's value, so that it can load it into the GRU.
> 
>     sgi-xp driver
> 
> The sgi-xp driver utilizes the node controller's message queue capability to
> send messages from one system partition (a single SSI) to another partition.
> 
> A message queue can be configured to have the node controller raise an
> interrupt whenever a message is written into it. This configuration is
> accomplished by setting up two processor writable MMRs located in the
> node controller. The vector number and apicid of the targeted CPU need
> to be written into one of these MMRs. There is no IOAPIC associated with
> this.
> 
> So one thought was that, once insmod'd, sgi-xp would allocate a message queue,
> allocate an irq/vector pair for a CPU located on the node where the message
> queue resides, and then set the MMRs with the memory address and length of the
> message queue and the vector and CPU's apicid. And then repeat, as there are
> actually two message queues required by this driver.

In addition to the above, the high resolution RTC timers in the UV hardware require that a vector be specified in order to send an interrupt to a specific destination when a timer expires.  The MMR's for these timers require a vector to be or'ed in with other values, including the interrupt's destination.  This is therefore done at run-time.

Like the GRU's vector, this vector is not in an IOAPIC.  This vector would be made available to all cpu's within a partition (SSI) and should be coupled with a per-cpu irq.

This is very similiar to what was available in earlier SGI hardware and used in drivers/char/mmtimer.c.