ARM PCI controller registration and representation using device tree?

ARM PCI controller registration and representation using device tree?

[Florian Fainelli]

Hi,

I was wondering if anyone had started working on representing the various PCI 
controllers found on SoCs to a device tree representation?

It seems like quite some generic code could be borrowed from PowerPC, 
especially the parsing of the PCI ranges, though I don't see ARM directly 
exposing a struct pci_controller to easily allow that.

Any thoughts about this?

Thanks.
-- 
Florian

ARM PCI controller registration and representation using device tree?

[Florian Fainelli]

Hi,

I was wondering if anyone had started working on representing the various PCI 
controllers found on SoCs to a device tree representation?

It seems like quite some generic code could be borrowed from PowerPC, 
especially the parsing of the PCI ranges, though I don't see ARM directly 
exposing a struct pci_controller to easily allow that.

Any thoughts about this?

Thanks.
-- 
Florian

ARM PCI controller registration and representation using device tree?

[Russell King - ARM Linux]

On Sat, Jun 02, 2012 at 04:18:11PM +0200, Florian Fainelli wrote:
> I was wondering if anyone had started working on representing the various PCI 
> controllers found on SoCs to a device tree representation?
> 
> It seems like quite some generic code could be borrowed from PowerPC, 
> especially the parsing of the PCI ranges, though I don't see ARM directly 
> exposing a struct pci_controller to easily allow that.
> 
> Any thoughts about this?

I think we have to keep the existing strategy of having struct hw_pci and
its pci_sys_data, registering each bus separately with its own individual
swizzle and map_irq functions depending on how the bus hardware is setup.
Also rememer that the PCI configuration space accessor functions are highly
PCI host bridge specific.

I don't think we can (or should) rely upon platforms having code in their
boot loader to probe the PCI bus, and provide the kernel with the PCI bus
structure.

I don't see this as a problem for DT - we just need a way to represent
each PCI bus as some kind of device.  I'm thinking whether we could get
away with converting the existing model to use platform devices, and
then we can use the existing DT platform device code to create the
relevant platform device.

Irrespective of whether your PCI host bridge appears as PCI device 0:0.0
or not, that's probably the right idea anyway.

Re: ARM PCI controller registration and representation using device tree?

[Russell King - ARM Linux]

On Sat, Jun 02, 2012 at 04:18:11PM +0200, Florian Fainelli wrote:
> I was wondering if anyone had started working on representing the various PCI 
> controllers found on SoCs to a device tree representation?
> 
> It seems like quite some generic code could be borrowed from PowerPC, 
> especially the parsing of the PCI ranges, though I don't see ARM directly 
> exposing a struct pci_controller to easily allow that.
> 
> Any thoughts about this?

I think we have to keep the existing strategy of having struct hw_pci and
its pci_sys_data, registering each bus separately with its own individual
swizzle and map_irq functions depending on how the bus hardware is setup.
Also rememer that the PCI configuration space accessor functions are highly
PCI host bridge specific.

I don't think we can (or should) rely upon platforms having code in their
boot loader to probe the PCI bus, and provide the kernel with the PCI bus
structure.

I don't see this as a problem for DT - we just need a way to represent
each PCI bus as some kind of device.  I'm thinking whether we could get
away with converting the existing model to use platform devices, and
then we can use the existing DT platform device code to create the
relevant platform device.

Irrespective of whether your PCI host bridge appears as PCI device 0:0.0
or not, that's probably the right idea anyway.

ARM PCI controller registration and representation using device tree?

[Arnd Bergmann]

On Saturday 02 June 2012, Russell King - ARM Linux wrote:
> I think we have to keep the existing strategy of having struct hw_pci and
> its pci_sys_data, registering each bus separately with its own individual
> swizzle and map_irq functions depending on how the bus hardware is setup.

Actually, I think we won't need to call pci_fixup_irqs for DT at all, so
there won't be host specific swizzle and map_irq functions. AFAICT the
interrupt-map property can handle all possible cases, it certainly does
so for a large number of obscure buses on powerpc.

Please have a look at drivers/of/of_pci_irq.c to see if you can spot
anything missing in there. The interrupt-map lets you define a mapping
from a bus-specific interrupt number to a global one, with an address/mask
of the bus/dev/fn ID of the device(s) it applies to. For anything with
default swizzling, we just provide the map for the root dev, while those
devices that require a more complex mapping need to list each IRQ line
that is connected differently. See arch/powerpc/boot/dts/mpc5121ads.dts
for an example with a nonstandard mapping.

> Also rememer that the PCI configuration space accessor functions are highly
> PCI host bridge specific.

Right. We can have a default ones for mmconfig and PIO 0xcfc in drivers/pci,
but I think most will have a separate one on ARM.

	Arnd

Re: ARM PCI controller registration and representation using device tree?

[Arnd Bergmann]

On Saturday 02 June 2012, Russell King - ARM Linux wrote:
> I think we have to keep the existing strategy of having struct hw_pci and
> its pci_sys_data, registering each bus separately with its own individual
> swizzle and map_irq functions depending on how the bus hardware is setup.

Actually, I think we won't need to call pci_fixup_irqs for DT at all, so
there won't be host specific swizzle and map_irq functions. AFAICT the
interrupt-map property can handle all possible cases, it certainly does
so for a large number of obscure buses on powerpc.

Please have a look at drivers/of/of_pci_irq.c to see if you can spot
anything missing in there. The interrupt-map lets you define a mapping
from a bus-specific interrupt number to a global one, with an address/mask
of the bus/dev/fn ID of the device(s) it applies to. For anything with
default swizzling, we just provide the map for the root dev, while those
devices that require a more complex mapping need to list each IRQ line
that is connected differently. See arch/powerpc/boot/dts/mpc5121ads.dts
for an example with a nonstandard mapping.

> Also rememer that the PCI configuration space accessor functions are highly
> PCI host bridge specific.

Right. We can have a default ones for mmconfig and PIO 0xcfc in drivers/pci,
but I think most will have a separate one on ARM.

	Arnd

ARM PCI controller registration and representation using device tree?

[Russell King - ARM Linux]

On Sun, Jun 03, 2012 at 02:41:52AM +0000, Arnd Bergmann wrote:
> On Saturday 02 June 2012, Russell King - ARM Linux wrote:
> > I think we have to keep the existing strategy of having struct hw_pci and
> > its pci_sys_data, registering each bus separately with its own individual
> > swizzle and map_irq functions depending on how the bus hardware is setup.
> 
> Actually, I think we won't need to call pci_fixup_irqs for DT at all, so
> there won't be host specific swizzle and map_irq functions. AFAICT the
> interrupt-map property can handle all possible cases, it certainly does
> so for a large number of obscure buses on powerpc.

That means you have to list every PCI device in DT along with its
interrupt number - which means your boot loader will have to scan the
PCI bus and provide interrupt and BAR settings into the kernel.

And that will suck if you have something like a 4 port PCI network card
which has a PCI-to-PCI bridge on - are boot loaders going to have the
correct swizzling support?  I think not.

Re: ARM PCI controller registration and representation using device tree?

[Russell King - ARM Linux]

On Sun, Jun 03, 2012 at 02:41:52AM +0000, Arnd Bergmann wrote:
> On Saturday 02 June 2012, Russell King - ARM Linux wrote:
> > I think we have to keep the existing strategy of having struct hw_pci and
> > its pci_sys_data, registering each bus separately with its own individual
> > swizzle and map_irq functions depending on how the bus hardware is setup.
> 
> Actually, I think we won't need to call pci_fixup_irqs for DT at all, so
> there won't be host specific swizzle and map_irq functions. AFAICT the
> interrupt-map property can handle all possible cases, it certainly does
> so for a large number of obscure buses on powerpc.

That means you have to list every PCI device in DT along with its
interrupt number - which means your boot loader will have to scan the
PCI bus and provide interrupt and BAR settings into the kernel.

And that will suck if you have something like a 4 port PCI network card
which has a PCI-to-PCI bridge on - are boot loaders going to have the
correct swizzling support?  I think not.

ARM PCI controller registration and representation using device tree?

[Arnd Bergmann]

On Sunday 03 June 2012, Russell King - ARM Linux wrote:
> 
> On Sun, Jun 03, 2012 at 02:41:52AM +0000, Arnd Bergmann wrote:
> > On Saturday 02 June 2012, Russell King - ARM Linux wrote:
> > > I think we have to keep the existing strategy of having struct hw_pci and
> > > its pci_sys_data, registering each bus separately with its own individual
> > > swizzle and map_irq functions depending on how the bus hardware is setup.
> > 
> > Actually, I think we won't need to call pci_fixup_irqs for DT at all, so
> > there won't be host specific swizzle and map_irq functions. AFAICT the
> > interrupt-map property can handle all possible cases, it certainly does
> > so for a large number of obscure buses on powerpc.
> 
> That means you have to list every PCI device in DT along with its
> interrupt number - which means your boot loader will have to scan the
> PCI bus and provide interrupt and BAR settings into the kernel.

Actually not, there is only one "interrupt-map" property in the PCI
host node, no need to list the add-on cards that use regular swizzling.

The binding is a little hard to understand at first, but it's all in there:
http://www.openfirmware.org/1275/proposals/Attachments/321att.pdf

> And that will suck if you have something like a 4 port PCI network card
> which has a PCI-to-PCI bridge on - are boot loaders going to have the
> correct swizzling support?  I think not.

I can assure you that those cases are covered by the binding. We have PCI
buses on most PowerPC systems, and we don't normally list the devices on
them, although the PCI binding allows us to. In the example you mention,
of_irq_map_pci will walk up the buses across all PCI bridges using standard
swizzling until it gets to a device that has a device node, which is
normally the host controller, but it can also be a device that is hardwired.

If a device or slot is hardwired to a specific IRQ, there is no problem
listing it in the device tree without the need for a bus walk in the boot
loarder. If it's not hardwired, then it should use the swizzling algorithm
in the bridge.

	Arnd

Re: ARM PCI controller registration and representation using device tree?

[Arnd Bergmann]

On Sunday 03 June 2012, Russell King - ARM Linux wrote:
> 
> On Sun, Jun 03, 2012 at 02:41:52AM +0000, Arnd Bergmann wrote:
> > On Saturday 02 June 2012, Russell King - ARM Linux wrote:
> > > I think we have to keep the existing strategy of having struct hw_pci and
> > > its pci_sys_data, registering each bus separately with its own individual
> > > swizzle and map_irq functions depending on how the bus hardware is setup.
> > 
> > Actually, I think we won't need to call pci_fixup_irqs for DT at all, so
> > there won't be host specific swizzle and map_irq functions. AFAICT the
> > interrupt-map property can handle all possible cases, it certainly does
> > so for a large number of obscure buses on powerpc.
> 
> That means you have to list every PCI device in DT along with its
> interrupt number - which means your boot loader will have to scan the
> PCI bus and provide interrupt and BAR settings into the kernel.

Actually not, there is only one "interrupt-map" property in the PCI
host node, no need to list the add-on cards that use regular swizzling.

The binding is a little hard to understand at first, but it's all in there:
http://www.openfirmware.org/1275/proposals/Attachments/321att.pdf

> And that will suck if you have something like a 4 port PCI network card
> which has a PCI-to-PCI bridge on - are boot loaders going to have the
> correct swizzling support?  I think not.

I can assure you that those cases are covered by the binding. We have PCI
buses on most PowerPC systems, and we don't normally list the devices on
them, although the PCI binding allows us to. In the example you mention,
of_irq_map_pci will walk up the buses across all PCI bridges using standard
swizzling until it gets to a device that has a device node, which is
normally the host controller, but it can also be a device that is hardwired.

If a device or slot is hardwired to a specific IRQ, there is no problem
listing it in the device tree without the need for a bus walk in the boot
loarder. If it's not hardwired, then it should use the swizzling algorithm
in the bridge.

	Arnd

ARM PCI controller registration and representation using device tree?

[Rob Herring]

On 06/02/2012 09:18 AM, Florian Fainelli wrote:
> Hi,
> 
> I was wondering if anyone had started working on representing the various PCI 
> controllers found on SoCs to a device tree representation?

Reusing the existing PCI bindings is the right way to go.

> It seems like quite some generic code could be borrowed from PowerPC, 
> especially the parsing of the PCI ranges, though I don't see ARM directly 
> exposing a struct pci_controller to easily allow that.

The PowerPC PCI DT code needing to be re-factored to generic code was my
conclusion as well in my brief look at it.

Rob

> 
> Any thoughts about this?
> 
> Thanks.

Re: ARM PCI controller registration and representation using device tree?

[Rob Herring]

On 06/02/2012 09:18 AM, Florian Fainelli wrote:
> Hi,
> 
> I was wondering if anyone had started working on representing the various PCI 
> controllers found on SoCs to a device tree representation?

Reusing the existing PCI bindings is the right way to go.

> It seems like quite some generic code could be borrowed from PowerPC, 
> especially the parsing of the PCI ranges, though I don't see ARM directly 
> exposing a struct pci_controller to easily allow that.

The PowerPC PCI DT code needing to be re-factored to generic code was my
conclusion as well in my brief look at it.

Rob

> 
> Any thoughts about this?
> 
> Thanks.

ARM PCI controller registration and representation using device tree?

[Arnd Bergmann]

On Saturday 02 June 2012, Florian Fainelli wrote:
> I was wondering if anyone had started working on representing the various PCI 
> controllers found on SoCs to a device tree representation?
> 
> It seems like quite some generic code could be borrowed from PowerPC, 
> especially the parsing of the PCI ranges, though I don't see ARM directly 
> exposing a struct pci_controller to easily allow that.
> 
> Any thoughts about this?

I think the PCI host controller code should be a lot more generic than it
currently is. Each architecture has to provide quite a bit of infrastructure
and the differences are to a large part not related to the CPU architecture.

I'd love to get to the point where we can have host controllers defined
in drivers/pci/host/*.c in a way that is completely architecture independent.
Alpha, ia64, microblaze, mips, powerpc, tile and xtensa all have a structure
named "pci_controller" for doing this, but I think they are all different.
ARM has two structures: pci_sys_data (corresponds to pci_controller) and
hw_pci mostly to provide function pointers that are all the same for each
instance. arch/sh has a pci_channel that does the same thing and parisc
has pci_hba_data.

I don't think it's realistic to aim for completely unifying those structures
and implementations, but we can try to define a new architecture independent
abstraction that covers all the common parts and has a pointer the architecture
specific data, which can hold the more obscure things.

	Arnd

Re: ARM PCI controller registration and representation using device tree?

[Arnd Bergmann]

On Saturday 02 June 2012, Florian Fainelli wrote:
> I was wondering if anyone had started working on representing the various PCI 
> controllers found on SoCs to a device tree representation?
> 
> It seems like quite some generic code could be borrowed from PowerPC, 
> especially the parsing of the PCI ranges, though I don't see ARM directly 
> exposing a struct pci_controller to easily allow that.
> 
> Any thoughts about this?

I think the PCI host controller code should be a lot more generic than it
currently is. Each architecture has to provide quite a bit of infrastructure
and the differences are to a large part not related to the CPU architecture.

I'd love to get to the point where we can have host controllers defined
in drivers/pci/host/*.c in a way that is completely architecture independent.
Alpha, ia64, microblaze, mips, powerpc, tile and xtensa all have a structure
named "pci_controller" for doing this, but I think they are all different.
ARM has two structures: pci_sys_data (corresponds to pci_controller) and
hw_pci mostly to provide function pointers that are all the same for each
instance. arch/sh has a pci_channel that does the same thing and parisc
has pci_hba_data.

I don't think it's realistic to aim for completely unifying those structures
and implementations, but we can try to define a new architecture independent
abstraction that covers all the common parts and has a pointer the architecture
specific data, which can hold the more obscure things.

	Arnd