pci_enable_msix throws up error

pci_enable_msix throws up error

[Ravinandan Arakali]

Hi,
I am seeing the following problem with MSI/MSI-X.

Note: I am copying netdev since other network drivers use
this feature and somebody on the list could throw light.

Our 10G network card(Xframe II) supports MSI and MSI-X.
When I load/unload the driver with MSI support followed
by an attempt to load with MSI-X, I get the following
message from pci_enable_msix:

"Can't enable MSI-X.  Device already has an MSI vector assigned"

I seem to be doing the correct things when unloading the
MSI driver. Basically, I do free_irq() followed by pci_disable_msi().
Any idea what I am missing ?

Further analysis:
Looking at the code, the following check(when it finds a match) in
msi_lookup_vector(called by pci_enable_msix) seems to throw up this
message:
if (!msi_desc[vector] || msi_desc[vector]->dev != dev ||
    msi_desc[vector]->msi_attrib.type != type ||
    msi_desc[vector]->msi_attrib.default_vector != dev->irq)

pci_enable_msi, on successful completion will populate the
fields in msi_desc. But neither pci_disable_msi nor free_irq
seems to undo/unpopulate the msi_desc table.
Could this be the cause for the problem ?

Thanks,
Ravi

MSI-X support on AMD 8132 platforms ?

[Ravinandan Arakali]

Hi,
I was wondering if anybody has got MSI-X going on AMD 8132 platforms.
Our network card and driver support MSI-X and the combination works
fine on IA64 and xeon platforms. But on the 8132, the MSI-X vectors are
assigned(pci_enable_msix succeeds) but no interrupts get generated.
Note that with a different OS, MSI-X does work on 8132.

Any suggestions are appreciated.

Thanks,
Ravi

Re: MSI-X support on AMD 8132 platforms ?

[Andi Kleen]

"Ravinandan Arakali" <ravinandan.arakali@neterion.com> writes:

> I was wondering if anybody has got MSI-X going on AMD 8132 platforms.
> Our network card and driver support MSI-X and the combination works
> fine on IA64 and xeon platforms. But on the 8132, the MSI-X vectors are
> assigned(pci_enable_msix succeeds) but no interrupts get generated.

See erratum #78 in the AMD 8132 Specification update.
It doesn't support the MSI capability and there are no plans to fix that.

AFAIK the only way to get MSI on Opteron is on PCI Express.

> Note that with a different OS, MSI-X does work on 8132.

Are you sure?

-Andi

Re: MSI-X support on AMD 8132 platforms ?

[Petr Vandrovec]

Andi Kleen wrote:
> "Ravinandan Arakali" <ravinandan.arakali@neterion.com> writes:
> 
> 
>>I was wondering if anybody has got MSI-X going on AMD 8132 platforms.
>>Our network card and driver support MSI-X and the combination works
>>fine on IA64 and xeon platforms. But on the 8132, the MSI-X vectors are
>>assigned(pci_enable_msix succeeds) but no interrupts get generated.
> 
> 
> See erratum #78 in the AMD 8132 Specification update.
> It doesn't support the MSI capability and there are no plans to fix that.
> 
> AFAIK the only way to get MSI on Opteron is on PCI Express.

I do not think that erratum #78 is related to this - it is related to tunnel 
itself generating MSI - which is not needed in this case.

>>Note that with a different OS, MSI-X does work on 8132.
> 
> Are you sure?

Can you provide 'lspci -vvvxxx' output from AMD8132 bridge?  (esp. bytes 
0xF4-0xFF from config space of 1022:7458 devices)  By default dword at 0xF4 is 
0xA8000008, disabling MSI/MSI-X mapping -> hypertransport interrupts.  Changing 
this to 0xA8010008 should enable this translation (iff qword at 0xF8 is 
0x0000FEE00000), allowing MSI to work on respective secondary/subordinate 
busses.  Unfortunately kernel ignores these HT capabilities...
								Petr

RE: MSI-X support on AMD 8132 platforms ?

[Ravinandan Arakali]

Hi Petr,
Thanks for the suggestion.
What you pointed out was the source of the problem. After
enabling the  MSI-to-HTInterrupts(the value at offset 0xF6),
I now see interrupts and can ping in both MSI and MSI-X modes.

Ravi

-----Original Message-----
From: Petr Vandrovec [mailto:vandrove@vc.cvut.cz]
Sent: Tuesday, May 16, 2006 8:01 AM
To: Andi Kleen
Cc: ravinandan.arakali@neterion.com; Peter. Phan; Leonid Grossman;
linux-kernel@vger.kernel.org
Subject: Re: MSI-X support on AMD 8132 platforms ?


Andi Kleen wrote:
> "Ravinandan Arakali" <ravinandan.arakali@neterion.com> writes:
>
>
>>I was wondering if anybody has got MSI-X going on AMD 8132 platforms.
>>Our network card and driver support MSI-X and the combination works
>>fine on IA64 and xeon platforms. But on the 8132, the MSI-X vectors are
>>assigned(pci_enable_msix succeeds) but no interrupts get generated.
>
>
> See erratum #78 in the AMD 8132 Specification update.
> It doesn't support the MSI capability and there are no plans to fix that.
>
> AFAIK the only way to get MSI on Opteron is on PCI Express.

I do not think that erratum #78 is related to this - it is related to tunnel
itself generating MSI - which is not needed in this case.

>>Note that with a different OS, MSI-X does work on 8132.
>
> Are you sure?

Can you provide 'lspci -vvvxxx' output from AMD8132 bridge?  (esp. bytes
0xF4-0xFF from config space of 1022:7458 devices)  By default dword at 0xF4
is
0xA8000008, disabling MSI/MSI-X mapping -> hypertransport interrupts.
Changing
this to 0xA8010008 should enable this translation (iff qword at 0xF8 is
0x0000FEE00000), allowing MSI to work on respective secondary/subordinate
busses.  Unfortunately kernel ignores these HT capabilities...
								Petr