Re: PCI: MSI interrupts masked using prohibited method

[Matthew Wilcox <matthew@wil.cx>]

On Fri, Jul 25, 2008 at 03:29:18PM +0200, Michal Schmidt wrote:
> The interesting thing is that I can see Destination ID bits of MSI
> Message Address change correctly in lspci output. But the interrupt is
> still delivered load-balanced to all CPUs even though the Destination
> ID identifies the single CPU I asked for. It seems the device only
> takes the new Message Address setting into account when the MSI Enable
> bit in the Message Control register is changed from 0 to 1. I tested
> this by setting the MSI enable bit to 0 and then immediately back to 1
> at the end of io_apic_64.c:set_msi_irq_affinity().
> 
> Is this a permitted behaviour for the device? I couldn't find anything
> in the PCI specification that would mentioned it.

I don't think that's necessary.  However, the thought occurs that we
ought to disable MSI, then write the address, then re-enable MSI.  It
doesn't cause a problem at the moment because we don't change the
top 32 bits of the address (at least on any of my systems ..) but
theoretically if we were to use a 64-bit address, we would experience
MSIs being sent to an address that was a mixture of the top 32 bits of
the old address and the bottom 32 bits of the new address.

We definitely can already get tearing when we've written the lower
address register but not the data register yet (also true for MSIX, by
the way).  So we ought to fix this properly.

We have the problem that we might still get interrupts on the old
pin-based interrupt line (ie David's original problem).  I have a
feeling somebody needs to register a handler for the pin-based interrupt
to handle this.  One possibility would be for the MSI code to register a
handler that calls the driver's MSI handler.  I don't think that's a
good idea though -- the driver's MSI handler is able to make different
assumptions from the pin handler.  Do we want to make drivers register
an interrupt handler for the original interrupt number before they try
to set up MSI?  It's certainly not what the PCI spec people had in mind,
but they seem to have overlooked this problem.

Yuck.

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."