Re: Multiple MSI messages support

[Roland Dreier <rdreier@cisco.com>]

 > > If this is really off-topic here, I apologize first. But I cannot
 > > think a better place to ask this particular question.
 > > I understand that the current PCI subsystem or linux kernel (x86)
 > > supports only one message when MSI is enabled even for devices having
 > > multiple MSI messages. But why? Is this a limitation solely due to the
 > > OS or due to the x86 APIC?
 > > I know the current linux kernel (2.6.23) changed MSI message data
 > > format a little bit to support other architectures. But some older
 > > version (e.g. 2.6.18) defined a specific format for the MSI msg data
 > > in a way that 8 bits contain the irq number and the other 8 bits have
 > > the interrupt attributes, which is x86 specific. Why does the msg data
 > > need to contain the irq number? Here is my hypothetic explanation. The
 > > device writes the MSI msg data to the specified MSI msg address. And
 > > APIC uses the irq number in the msg data to generate appropriate
 > > interrupt, which of course results in an appropriate ISR invoked. A
 > > device having multiple MSI messages typically appends some information
 > > of which MSI message to the msg data field. For example, if the system
 > > (or OS) configures the MSI msg data as 0x5000, a device having 4 MSI
 > > messages could write 0x5000, 0x5001, 0x5002, 0x5003 to differentiate
 > > the MSI messages. However this cannot work with the APIC due to the
 > > way how APIC asserts interrupts as I described above (if my
 > > understanding is correct).
 > > Hence my answer to the question is this is due to the x86 APIC. For
 > > other architectures such as powerpc this is probably not a problem
 > > since the interrupt controller is different. Am I correct?

 > IMO it's more like there has never been enough need for anybody to
 > look into it, I bet...

Actually the original poster's explanation is pretty accurate.  MSI
imposes a restriction on how a device generates multiple messages,
since there is only one message data register and the rest of the
messages must be based on that in a simple way.  And the format of
"Intel-style" APIC MSI messages does not match up with the way the PCI
spec generates multiple messages.

In fact at least IBM pSeries boxes seem to be able to use MSI to
generate multiple interrupts from the same device -- BenH can probably
give details.

Multiple interrupt messages are supported by Linux via MSI-X (which
allows each message to have completely different data).

 > The way drivers are written, you are typically must touch a few key
 > hardware registers _anyway_, so the multiple messages in practice are
 > not much more useful than the simple fact that your MSI irq handler
 > function was called (with all that indicates and implies).

For high-performance devices this is not really true.  The InfiniBand
mthca driver can use MSI to get a single interrupt, or MSI-X to get
different interrupts for different types of events.  MSI-X allows the
read of the "interrupt cause register" to be avoided, and this ends up
making a measurable performance difference for the fast path.

Also, obviously, having multiple interrupts means you can bind
different interrupts to different CPUs, which will become very
important if/when things like multi-queue NICs are supported.

 - R.