Re: [RFC/PATCHv2] x86/irq: round-robin distribution of irqs to cpus w/in node

[Arthur Kepner <akepner@sgi.com>]

(Compendium reply to 2 emails.)

On Mon, Sep 27, 2010 at 05:17:07PM -0700, Eric W. Biederman wrote:
> Thomas Gleixner <tglx@linutronix.de> writes:
> 
> > On Mon, 27 Sep 2010, Arthur Kepner wrote:
> >
> ......
> The deep bug is that create_irq_nr allocates a vector (which it does
> because at the time there was no better way to mark an irq in use on
> x86).  In the case of msi-x we really don't know the node that irq is
> going to be used on until we get a request irq.  We simply know which
> node the device is on.
> 
> If you want to see what is going follow the call trace looks like.
> pci_enable_msix 
>   arch_setup_msi_irqs
>     create_irq_nr
> 
> After pci_enable_msix is finished then the driver goes and makes all
> of the irqs per cpu irqs.
> 
> There are goofy things that happen when hardware asks for 1 irq per cpu.
> But since msi can ask for up to 4096 irqs (assuming the hardware
> supports it) I can totally see putting all 256 of those irqs on a single
> cpu, before you go to user space and let user space or something
> reassign all of those irqs in a per cpu way.
> 

Yes, that's exactly the problem. All of the vectors on the lowest 
numbered CPUs get used. Any subsequent request for an interrupt on 
one of the low numbered CPUs will fail.

> .....

On Tue, Sep 28, 2010 at 03:59:33AM -0700, Eric W. Biederman wrote:
> Thomas Gleixner <tglx@linutronix.de> writes:
> 
> > On Mon, 27 Sep 2010, Eric W. Biederman wrote:
> >> > On Mon, 27 Sep 2010, Arthur Kepner wrote:
> >> The deep bug is that create_irq_nr allocates a vector (which it does
> >> because at the time there was no better way to mark an irq in use on
> >> x86).  In the case of msi-x we really don't know the node that irq is
> >> going to be used on until we get a request irq.  We simply know which
> >> node the device is on.
> >
> > Bah. So the whole per node allocation business is completely useless
> > at this point.
> 
> Probably.

Huh? No, the patch that started this thread spreads the irqs around 
and avoids the problem of a single CPU's vectors all being consumed.

> ...
> 
> Understood.  It has taken a couple of years before this bug finally
> bit anyone waiting a release or two to get it fixed properly seems
> reasonable.
> ....

And so what are we to do in the meantime? At the moment we're 
disabling MSIX, which is a pretty unattractive workaround.

-- 
Arthur