PCI MSI: the new interrupt routing headache

PCI MSI: the new interrupt routing headache

[Jeff Garzik]

I just got SATA working on Marvell.  After fixing a bunch of issues in 
the driver, the final issue was lack of interrupts.  Disabling 
CONFIG_PCI_MSI solved that, and suddenly the driver was working quite 
nicely.

The general problem is that pci_enable_msi() is not failing, on systems 
that do not support MSI.  This leads to Infiniband, tg3, and other 
drivers working around this problem by including an MSI-interrupts-work 
test during probe.

Perhaps its because I like leading edge stuff, and am playing with 
drivers for PCI MSI hardware, but it seems like I am running into this 
pci_enable_msi()-doesnt-fail problem more and more frequently.  First 
tg3, then AHCI, now sata_mv.

What needs to be done, to detect working PCI message signalled 
interrupts such that pci_enable_msi() fails properly?

Thanks,

	Jeff

Re: PCI MSI: the new interrupt routing headache

[Max Krasnyansky]

Hi Jeff,
> 
> I just got SATA working on Marvell.  After fixing a bunch of issues in 
> the driver, the final issue was lack of interrupts.  Disabling 
> CONFIG_PCI_MSI solved that, and suddenly the driver was working quite 
> nicely.
> 
> The general problem is that pci_enable_msi() is not failing, on systems 
> that do not support MSI.  This leads to Infiniband, tg3, and other 
> drivers working around this problem by including an MSI-interrupts-work 
> test during probe.
> 
> Perhaps its because I like leading edge stuff, and am playing with 
> drivers for PCI MSI hardware, but it seems like I am running into this 
> pci_enable_msi()-doesnt-fail problem more and more frequently.  First 
> tg3, then AHCI, now sata_mv.
> 
> What needs to be done, to detect working PCI message signalled 
> interrupts such that pci_enable_msi() fails properly?
It fails correctly when MSI quirk is enabled. I had the same problems
on AMD 8131 based machine with 2.6.11 kernel. E1000 was enabling MSI and
not getting any interrupts. Later (.13 I think) MSI quirk was introduced for
8131 chipset and everything started working (without having to hack E1000
that is).
May be you should just add your platform to the PCI quirks list ?

Max

Re: PCI MSI: the new interrupt routing headache

[Roland Dreier]

    Jeff> What needs to be done, to detect working PCI message
    Jeff> signalled interrupts such that pci_enable_msi() fails
    Jeff> properly?

There are two things that cause MSIs not to work.  First, the PCI host
bridge may not have working MSI support.  To handle this, we have the
"msi_quirk" which is set by the PCI quirk code.  For example, this is
used on systems with AMD-8131 PCI-X bridges.  (As I've noted
elsewhere, this is actually too crude a method -- actual systems exist
with e.g. both AMD-8131 and Nforce4 PCI bridges, so that MSI works for
PCIe devices but not devices below the AMD-8131)

Second, there are PCI devices that have an MSI capability but which
don't have working MSI support.  Most revisions of the e1000 fall into
this category.  In this case, it is up to the driver to know when it's
safe to try to enable MSI.

However, given that MSI/MSI-X is not in wide use, there is undoubtedly
much more broken hardware (both chipsets and devices) that we don't
know about and need to add quirks or driver workarounds for.

Hence, in the interest of discovering this hardware and also in
getting less cryptic bug reports, it's a good idea to add a test that
interrupts actually work in any driver that tries to enable MSI or
MSI-X.  Since it requires knowledge of a device to know how to get the
device to trigger an interrupt, this test has to be done in each
driver and can't be centralized.

 - R.