Re: MSIX and multiple reply queues

[Matthew Wilcox <willy@linux.intel.com>]

On Mon, Jun 21, 2010 at 06:38:25PM -0500, James Bottomley wrote:
> So I think what you're saying is that you plan to have one MSI-X vector
> per CPU (which is possbile)?  If so, you just bind the vector affinity
> of the interrupt to the CPU you want.  If it's something more complex
> than this, I'd suggest asking the PCI list ... I cc'd them in case they
> have any insight.

This is something I've been discussing with the Intel 10Gbit NIC people.
They want the same thing you do -- spread the interrupts out across the
different CPUs.  I think we need an API to have the PCI subsystem set up
as many MSI-X interrupts as possible (limited by # supported by device
and # of CPUs), and spread them out across the CPUs as widely as possible
(per logical CPU if we have enough, per core, per socket, even per node).
Then we need an API to go from CPU number to MSI-X vector number.

Something else I've been musing is the idea of marking these interrupts
as per-cpu (since, well, they are).  That gives an optmised interrupt
handler path in __do_IRQ.  It's going to make the ->set_affinity handler
significantly different, but it seems worth doing.

No idea when I'll have time to do this ...