MSI support on Linux PCI implementation for Ocotea

MSI support on Linux PCI implementation for Ocotea

[Shawn Jin]

Hi,

I'm looking at the linux PCI implementation, especially MSI support
for Ocotea. And I have some observations and questions about it. Maybe
somebody here can shed some light on them. Thanks.

1. Obviously MSI is supported in Linux 2.6.x, maybe even in 2.4.x. But
MSI implementation seems to support only IOxAPIC on x86 or IA64
architectures, though the implemenation is in generic drivers/pci
tree.

2. Why is the message data defined in a such way shown as follows? Or
does it just follow IOxAPIC's design? Further question is who defines
the format of the message data. The PCI/PCIe specs don't say anything
about the content of the message data register.

struct msg_data {
	__u32	vector		:  8;
	__u32	delivery_mode	:  3;	/* 000b: FIXED | 001b: lowest prior */
	__u32	reserved_1	:  3;
	__u32	level		:  1;	/* 0: deassert | 1: assert */
	__u32	trigger		:  1;	/* 0: edge | 1: level */
	__u32	reserved_2	: 16;
} __attribute__ ((packed));

3. How is the pci subsystem initialized on a PPC platform? Is there
any existing document discussing PCI subsystem? Of course I've been
doing a lot internet search and couldn't find too much useful infor.

Thanks a lot,
-Shawn.

Re: MSI support on Linux PCI implementation for Ocotea

[Benjamin Herrenschmidt]

On Thu, 2006-06-15 at 14:14 -0700, Shawn Jin wrote:
> Hi,
> 
> I'm looking at the linux PCI implementation, especially MSI support
> for Ocotea. And I have some observations and questions about it. Maybe
> somebody here can shed some light on them. Thanks.

Hi ! What is Ocotea ! :)

> 1. Obviously MSI is supported in Linux 2.6.x, maybe even in 2.4.x. But
> MSI implementation seems to support only IOxAPIC on x86 or IA64
> architectures, though the implemenation is in generic drivers/pci
> tree.

Exact. And it's also a pile of crap.

> 2. Why is the message data defined in a such way shown as follows? Or
> does it just follow IOxAPIC's design? Further question is who defines
> the format of the message data. The PCI/PCIe specs don't say anything
> about the content of the message data register.

 .../...

It's all intel specific junk.

> 3. How is the pci subsystem initialized on a PPC platform? Is there
> any existing document discussing PCI subsystem? Of course I've been
> doing a lot internet search and couldn't find too much useful infor.

pci host bridges are initialized early, generally in setup_arch() and
the PCI bus is probed later. You can see the code in
arch/powerpc/kernel/pci_{32,64}.c

Now, regarding MSI, as you have noticed, the situation isn't great. I'm
currently in the middle of reworking our interrupt management (and
interrupt numbers allocation layer) and Michael Ellermann is looking at
the MSI issue in parallel.

We don't have a solution yet but are working on it.

Ben.

Re: MSI support on Linux PCI implementation for Ocotea

[Roland Dreier]

    Benjamin> Hi ! What is Ocotea ! :)

It's an eval board for some PowerPC 440 ... 440GX I think (which has
the HW capability to support MSI on PCI-X iirc)

 - R.

Re: MSI support on Linux PCI implementation for Ocotea

[Shawn Jin]

> Hi ! What is Ocotea ! :)

Did I spell it wrong?? I don't believe Ben doesn't know Ocotea.
Hmmm...Ocotea again? Let me check if I really spelled it wrong. Hey,
it's really Ocotea. ;)

Of course, there is no such MSI support specific for Ocotea.

> > 1. Obviously MSI is supported in Linux 2.6.x, maybe even in 2.4.x. But
> > MSI implementation seems to support only IOxAPIC on x86 or IA64
> > architectures, though the implemenation is in generic drivers/pci
> > tree.
>
> Exact. And it's also a pile of crap.

;) I'm not in a position where I can either agree or disagree with
you. :D But may I ask if ppc platform can make use of this crap in our
PCI express testing? That is, are the functionalities of this crap
working?

> pci host bridges are initialized early, generally in setup_arch() and
> the PCI bus is probed later. You can see the code in
> arch/powerpc/kernel/pci_{32,64}.c
>
> Now, regarding MSI, as you have noticed, the situation isn't great. I'm
> currently in the middle of reworking our interrupt management (and
> interrupt numbers allocation layer) and Michael Ellermann is looking at
> the MSI issue in parallel.
>
> We don't have a solution yet but are working on it.

It's good to know the development status. What's the relationship
between generic pci driver layer in drivers/pci and architecture
specific implementation? One thing I notice is that architectures
provide pcibios_init() and do resource allocation while the generic
driver provides PCI generic functions such as pci_scan_device().

Thanks,
-Shawn.

Re: MSI support on Linux PCI implementation for Ocotea

[Shawn Jin]

Hi Ben,

> Now, regarding MSI, as you have noticed, the situation isn't great. I'm
> currently in the middle of reworking our interrupt management (and
> interrupt numbers allocation layer) and Michael Ellermann is looking at
> the MSI issue in parallel.

Just one point about MSI(-X) message address and data format. I think
it should be the platform that defines the content. Current
implementation doesn't have this flexibility. We have a design where
MSI mode uses message data to carry vector number info while MSI-X
mode uses message address to carry this info. Although it seems
strange, but I think it's allowed by the spec.

Thanks,
-Shawn.

Re: MSI support on Linux PCI implementation for Ocotea

[Benjamin Herrenschmidt]

On Fri, 2006-06-16 at 14:45 -0700, Shawn Jin wrote:
> Hi Ben,
> 
> > Now, regarding MSI, as you have noticed, the situation isn't great. I'm
> > currently in the middle of reworking our interrupt management (and
> > interrupt numbers allocation layer) and Michael Ellermann is looking at
> > the MSI issue in parallel.
> 
> Just one point about MSI(-X) message address and data format. I think
> it should be the platform that defines the content.

That's an obvious requirement :)

> Current
> implementation doesn't have this flexibility. We have a design where
> MSI mode uses message data to carry vector number info while MSI-X
> mode uses message address to carry this info. Although it seems
> strange, but I think it's allowed by the spec.

The current implementation is bad :) There are some patches for altix
that are in -mm that add some platform hooks that make the situation a
little bit better, but it's still far from perfect.

Ben.