Re: [PATCH v2] PCI: Mark broken INTx masking for Mellanox devices

[Gavin Shan <gwshan@linux.vnet.ibm.com>]

On Sun, Aug 03, 2014 at 09:08:06AM -0600, Alex Williamson wrote:
>On Mon, 2014-08-04 at 00:30 +1000, Gavin Shan wrote:
>> On Sun, Aug 03, 2014 at 08:57:39AM +0000, Eli Cohen wrote:
>> >>>
>> >>>What is the problem with masking the interrupts with the PCI command 
>> >>>register? I'm asking because I want to understand in which devices we 
>> >>>have the problem, and if it could be fixed by firmware guys.
>> >>>What are the implications of having the quirk?
>> >>>
>> >
>> >>The way to mask the interrupt through PCI command register isn't taking effect on IBM power platform. So we have to have the >quirk so that the interrupt could be masked from interrupt controller side with function disable_irq_nosync().
>> >>
>> >>If the interrupt can't be masked properly, we detect interrupt storm reported from host/guest when passing through those devices >via VFIO without suprise.
>> >
>> >Hi Gavin,
>> >Does it have any effect on performance. Also, can you tell in which cases interrupts need to be masked?
>> >
>> 
>> Eli, more code needed to be run for masking the LSI from interrupt controller
>> side than from PCI command register.
>> 
>> I was passing through Mellanox devices from host to guest with VFIO, and I
>> designated to use LSI in the guest side. More details could be found in
>> drivers/vfio/pci/vfio_pci_intrs.c::vfio_intx_handler()
>
>INTx is relatively high overhead already for device assignment since the
>interrupt is level triggered and needs to be masked on the host while
>the guest is processing it.  The more important restriction imposed by
>marking broken INTx masking is that the device needs an exclusive
>interrupt line in order to be assigned to a guest.  That may be common
>practice on IBM power, but on x86 it can make it much harder to
>configure the system for this use case.  Thanks,
>

Power platform has the similar situation: Each PCI controller has 4
LSIs shared by all child devices attached to the PHB. It would be
racy if one LSI is shared by 2 or more devices. So masking LSI with
PCI command register is the preferred mechanism. Unfortunately, it
doesn't work on those 2 Mellanox devices. With the quirk, it's workable
at least.

Thanks,
Gavin

>Alex
>
>