Is there a way to change irq numbers of pcie cards ?

Is there a way to change irq numbers of pcie cards ?

[Bert dd]

Hi,
Is there a way to change irq numbers of pcie cards ?

I have a supermicro machine with 2 raid cards (lsi 9260-8i and lsi 9260-8e) and
one nvidia GTX480 graphics card.
The problem is that all these pcie cards are assigned the same irq number,
and thus all the interrupts are routed to the same cpu which can not
handle this.
If I replace the nvidia GTX480 with an older graphics card(too old for
our purposes (GTX295)), the graphics card gets a different irq number
than the raid cards, the interrups of the graphics card are handled by
a different cpu than those of the raid cards, and the read/write speed
of the raid increases.
I am reading/writing at 2.5 GB/s to the raid and I am uploading
textures at 0.5 GB/s to the graphics hardware.
Is there a way to change the irq numbers of the pcie cards ?
I am using IOAPIC for interrupt routing.
I am using ubuntu - kernel version 2.6.31-14-generic.

thanks,
  Bert De Decker
  bert.dedecker@uhasselt.be

Re: Is there a way to change irq numbers of pcie cards ?

[Bjorn Helgaas]

[I added linux-pci, where more PCI-savvy people will see it.]

On Wednesday, June 02, 2010 06:08:33 am Bert dd wrote:
> Hi,
> Is there a way to change irq numbers of pcie cards ?
> 
> I have a supermicro machine with 2 raid cards (lsi 9260-8i and lsi 9260-8e) and
> one nvidia GTX480 graphics card.
> The problem is that all these pcie cards are assigned the same irq number,
> and thus all the interrupts are routed to the same cpu which can not
> handle this.
> If I replace the nvidia GTX480 with an older graphics card(too old for
> our purposes (GTX295)), the graphics card gets a different irq number
> than the raid cards, the interrups of the graphics card are handled by
> a different cpu than those of the raid cards, and the read/write speed
> of the raid increases.
> I am reading/writing at 2.5 GB/s to the raid and I am uploading
> textures at 0.5 GB/s to the graphics hardware.
> Is there a way to change the irq numbers of the pcie cards ?
> I am using IOAPIC for interrupt routing.

I would think performance-oriented PCIe devices would be using
some sort of MSI, but I'm not really an expert in that area.

Can you post your /proc/interrupts and complete dmesg log in
case nobody jumps in with an obvious answer?

Bjorn

> I am using ubuntu - kernel version 2.6.31-14-generic.
> 
> thanks,
>   Bert De Decker
>   bert.dedecker@uhasselt.be

Re: Is there a way to change irq numbers of pcie cards ?

[Robert Hancock]

On 06/02/2010 06:08 AM, Bert dd wrote:
> Hi,
> Is there a way to change irq numbers of pcie cards ?
>
> I have a supermicro machine with 2 raid cards (lsi 9260-8i and lsi 9260-8e) and
> one nvidia GTX480 graphics card.
> The problem is that all these pcie cards are assigned the same irq number,
> and thus all the interrupts are routed to the same cpu which can not
> handle this.
> If I replace the nvidia GTX480 with an older graphics card(too old for
> our purposes (GTX295)), the graphics card gets a different irq number
> than the raid cards, the interrups of the graphics card are handled by
> a different cpu than those of the raid cards, and the read/write speed
> of the raid increases.
> I am reading/writing at 2.5 GB/s to the raid and I am uploading
> textures at 0.5 GB/s to the graphics hardware.
> Is there a way to change the irq numbers of the pcie cards ?
> I am using IOAPIC for interrupt routing.
> I am using ubuntu - kernel version 2.6.31-14-generic.

In general there's no way to change what interrupts get allocated to 
what slots, unless the BIOS has a setting somewhere to change it.

If you can get MSI enabled for some of those cards, though, that should 
prevent them from sharing interrupts. What driver are you using for the 
nvidia cards? You can turn on MSI on the binary nvidia driver with the 
NVreg_EnableMSI=1 module parameter..

Re: Is there a way to change irq numbers of pcie cards ?

[Clemens Ladisch]

Robert Hancock wrote:
> On 06/02/2010 06:08 AM, Bert dd wrote:
> > I have a supermicro machine with 2 raid cards (lsi 9260-8i and lsi 9260-8e) and
> > one nvidia GTX480 graphics card.
> > The problem is that all these pcie cards are assigned the same irq number,
> 
> If you can get MSI enabled for some of those cards, though, that should
> prevent them from sharing interrupts.

The PCIe spec says that all PCIe devices must support MSI interrupts.
(Whether they do in practice is another question ...)

Regardless of the graphics card, enabling MSI for the RAID cards will
prevent their interrupts from interfering with each other.
Bert, you might want to try this completely untested patch:


--- a/drivers/scsi/megaraid/megaraid_sas.c
+++ b/drivers/scsi/megaraid/megaraid_sas.c
@@ -3169,6 +3169,8 @@ megasas_probe_one(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (megasas_init_mfi(instance))
 		goto fail_init_mfi;
 
+	pci_enable_msi(pdev);
+
 	/*
 	 * Register IRQ
 	 */
@@ -3218,6 +3220,7 @@ megasas_probe_one(struct pci_dev *pdev, const struct pci_device_id *id)
 	pci_set_drvdata(pdev, NULL);
 	instance->instancet->disable_intr(instance->reg_set);
 	free_irq(instance->pdev->irq, instance);
+	pci_disable_msi(pdev);
 
 	megasas_release_mfi(instance);
 
@@ -3354,6 +3357,7 @@ megasas_suspend(struct pci_dev *pdev, pm_message_t state)
 	pci_set_drvdata(instance->pdev, instance);
 	instance->instancet->disable_intr(instance->reg_set);
 	free_irq(instance->pdev->irq, instance);
+	pci_disable_msi(pdev);
 
 	pci_save_state(pdev);
 	pci_disable_device(pdev);
@@ -3416,6 +3420,8 @@ megasas_resume(struct pci_dev *pdev)
 	tasklet_init(&instance->isr_tasklet, megasas_complete_cmd_dpc,
 			(unsigned long)instance);
 
+	pci_enable_msi(pdev);
+
 	/*
 	 * Register IRQ
 	 */
@@ -3519,6 +3525,7 @@ static void __devexit megasas_detach_one(struct pci_dev *pdev)
 	instance->instancet->disable_intr(instance->reg_set);
 
 	free_irq(instance->pdev->irq, instance);
+	pci_disable_msi(pdev);
 
 	megasas_release_mfi(instance);