PCI Express MSI support on Kirkwood?

PCI Express MSI support on Kirkwood?

[Lennert Buytenhek]

On Wed, Apr 21, 2010 at 05:55:27AM -0700, Leon Woestenberg wrote:

> > what is the current status for PCI Express MSI support on Kirkwood?
> 
> Including Nicolas Pitre, Lennert Buytenhek and LAK ml on this topic.
> 
> I'm interested in PCI Express MSI support for Linux / Kirkwood, has
> this been addressed yet in any of the GIT trees?

It hasn't been done yet as far as I know.

I don't think KW was designed with MSI in mind, but it can perhaps be
done (MSI with up to 16 messages but without multiple message enable,
or MSI-X with up to 32 system-wide messages) if you use the interrupt
controller Doorbell interrupts for this purpose.

IOW, make the KW 1M register area available to the PCIe bus via an
inbound BAR, point the MSI message address to the Host-to-CPU Doorbell
Register (register block + 0x20400), and use successive powers of two
for the MSI message data.  (And demultiplex the doorbell interrupt into
its own range of 32 IRQs in the Kirkwood IRQ demux code.)

Having to map the register area to an inbound BAR means that it's not
possible anymore to have more than 2G of RAM in the system.  On KW
this won't be an issue, but on mv78xx0 it would be in some
configurations.

/me doesn't think he has MSI-capable PCIe cards

PCI Express MSI support on Kirkwood?

[Leon Woestenberg]

Hello Lennert,

On Wed, Apr 21, 2010 at 3:14 PM, Lennert Buytenhek
<buytenh@wantstofly.org> wrote:
> On Wed, Apr 21, 2010 at 05:55:27AM -0700, Leon Woestenberg wrote:
>>
>> I'm interested in PCI Express MSI support for Linux / Kirkwood, has
>> this been addressed yet in any of the GIT trees?
>
> It hasn't been done yet as far as I know.
>
> I don't think KW was designed with MSI in mind, but it can perhaps be
> done (MSI with up to 16 messages but without multiple message enable,
> or MSI-X with up to 32 system-wide messages) if you use the interrupt
> controller Doorbell interrupts for this purpose.
>
The datasheet mentions support for Root Complex mode as well, but does
not any specific address for the
Message Address Register, so that would mean I could use any
(allocated) piece of system RAM, for example?

"6.1.1 Message Signaled Interrupts (MSI)
Message Signaled Interrupts (MSI) are supported in both Root Complex
and Endpoint modes.
Root Complex mode: The Host sets the PCI Express MSI Message Address Register
                      (Table 288 p. 461) to the same value that it has
set the Endpoint device.
                      A memory write received, with the same address,
is handle as an MSI.
                      Upon receipt of MSI, an interrupt is set in the
<RcvMsi> field in the PCI
                      Express Interrupt Cause Register (Table 320 p.
487). Interrupt data is
                      saved in the PCI Express MSI Message Data Register
                      (Table 290 p. 461).
"



> IOW, make the KW 1M register area available to the PCIe bus via an
> inbound BAR,<...>
>
I wonder if this is necessary after reading 6.1.1 (see above), can you
comment on this?

Do you mean using the BAR0:

"The PCI Express port has its own address map. The PCI Express
interface address map consists of
three Base Address registers (BARs) that map the chip address space.
One BAR is dedicated for
the chip internal registers space."

>
> /me doesn't think he has MSI-capable PCIe cards
>
I have used a cheap Intel PCI Express x1 E1000 Ethernet controller to
verify MSI on several embedded systems before working with custom
hardware.

The E1000 chip works with MSI, if such support is available on the
platform, falls back to legacy interrupts, if not.

Regards,
-- 
Leon

PCI Express MSI support on Kirkwood?

[Lennert Buytenhek]

On Wed, Apr 28, 2010 at 11:34:14PM +0200, Leon Woestenberg wrote:

> Hello Lennert,

Hello Leon,

Removing the openrd@ group from the CC as it's a closed group.
Removing nico at marvell.com from the CC as Nico is no longer with Marvell.


> >> I'm interested in PCI Express MSI support for Linux / Kirkwood, has
> >> this been addressed yet in any of the GIT trees?
> >
> > It hasn't been done yet as far as I know.
> >
> > I don't think KW was designed with MSI in mind, but it can perhaps be
> > done (MSI with up to 16 messages but without multiple message enable,
> > or MSI-X with up to 32 system-wide messages) if you use the interrupt
> > controller Doorbell interrupts for this purpose.
> 
> The datasheet mentions support for Root Complex mode as well, but does
> not any specific address for the Message Address Register,

The Root Complex doesn't need to have an MSI or MSI-X capability
structure -- it's a property of the endpoint (which generates
interrupts).


> so that would mean I could use any (allocated) piece of system RAM,
> for example?

Yes, but the whole point is that the write to the special MSI
address will generate an interrupt to the host CPU, while if you fill
in a PCI bus address corresponding to system RAM, you'd then have to
poll that RAM location to see if interrupts occured.

(Be careful when choosing the MSI address -- if any MSI write is
terminated with an abort, you end up triggering SERR#.)


> "6.1.1 Message Signaled Interrupts (MSI)
> Message Signaled Interrupts (MSI) are supported in both Root Complex
> and Endpoint modes.
> Root Complex mode: The Host sets the PCI Express MSI Message Address Register
>                       (Table 288 p. 461) to the same value that it has
> set the Endpoint device.
>                       A memory write received, with the same address,
> is handle as an MSI.
>                       Upon receipt of MSI, an interrupt is set in the
> <RcvMsi> field in the PCI
>                       Express Interrupt Cause Register (Table 320 p.
> 487). Interrupt data is
>                       saved in the PCI Express MSI Message Data Register
>                       (Table 290 p. 461).
> "

Where is this from?  (This doesn't appear in my copy of the Kirkwood
functional spec at least.)

The address match described here is cute, but the fact that it saves
the data in a single register appears to make it kind of useless for
use with multiple interrupt sources.  (You're likely still better off
with the doorbell registers, as they can collapse multiple events and
will never overflow.)