MPC8536 PCI rescan to discover FPGA

MPC8536 PCI rescan to discover FPGA

[Felix Radensky]

Hi,

On a custom MPC8536 board running linux-2.6.31,
I'd like to load FPGA code from linux and then rescan
PCI-E bus to discover FPGA device. Is that possible ?
When linux boots FPGA is not loaded, so initial PCI
scan does not detect it.

I've tried playing with /sys/bus/pci/rescan and 
/sys/bus/pci/devices/.../rescan
but didn't have much success.

Thanks.

Felix.

Re: MPC8536 PCI rescan to discover FPGA

[David Hawkins]

Hi Felix,

> On a custom MPC8536 board running linux-2.6.31,
> I'd like to load FPGA code from linux and then rescan
> PCI-E bus to discover FPGA device. Is that possible ?
> When linux boots FPGA is not loaded, so initial PCI
> scan does not detect it.
> 
> I've tried playing with /sys/bus/pci/rescan and 
> /sys/bus/pci/devices/.../rescan
> but didn't have much success.

This can be made to work using the kernel hot-swap
interface. PCI devices have an ENUM# interrupt that
they assert when inserted or extracted, and the host
hot-swap driver can be hooked up to it. PCI-E may
have a similar mechanism, if it does, then when your
FPGA configures as a PCI-E device, it can assert that
interrupt line (or send the appropriate PCI-E
message to simulate that interrupt).

However, even if PCI-E does not have the concept of
an ENUM# interrupt there is a way to generate a fake
hot-swap event and generate a re-scan of the PCI bus.

I haven't tested the kernel hot-swap interface, but I
know that Ira did, so I'll cc him on this mail, and he
can let you know what he tested.

Cheers,
Dave

Re: MPC8536 PCI rescan to discover FPGA

[Benjamin Herrenschmidt]

On Mon, 2009-09-21 at 09:04 -0700, David Hawkins wrote:
> This can be made to work using the kernel hot-swap
> interface. PCI devices have an ENUM# interrupt that
> they assert when inserted or extracted, and the host
> hot-swap driver can be hooked up to it. PCI-E may
> have a similar mechanism, if it does, then when your
> FPGA configures as a PCI-E device, it can assert that
> interrupt line (or send the appropriate PCI-E
> message to simulate that interrupt).
> 
> However, even if PCI-E does not have the concept of
> an ENUM# interrupt there is a way to generate a fake
> hot-swap event and generate a re-scan of the PCI bus.
> 
> I haven't tested the kernel hot-swap interface, but I
> know that Ira did, so I'll cc him on this mail, and he
> can let you know what he tested.

Right. However, in case it's a bit too much work to get
hotswap implemented on the machine, you may still be able
to do something simpler from your platform code, after you've
finished loading the FPGA. I assume the FPGA doesn't contain a
P2P bridge that would require probing further below the FPGA
itself.

The basic idea is to call pci_scan_slot() on the devfn where
the FPGA is supposed to respond.

Then you need to also do some fixup. First you need to call
pcibios_setup_bus_devices(). This will wire up the device
to an OF node if you have one, setup some default DMA ops,
etc...

Note that this function will walk over all devices on that bus
which is interesting since some of those may have already been
fully setup initially. Hopefully that isn't a problem. If it
was to become one, we would have to figure out a way to skip
devices that have already been "setup".

And finally you call pcibios_finish_adding_to_bus() which will
do the resource allocation pass on all new devices on the bus
and register them with the core device layer.

Cheers,
Ben.

Re: MPC8536 PCI rescan to discover FPGA

[Felix Radensky]

Hi, Benjamin

Benjamin Herrenschmidt wrote:
> Right. However, in case it's a bit too much work to get
> hotswap implemented on the machine, you may still be able
> to do something simpler from your platform code, after you've
> finished loading the FPGA. I assume the FPGA doesn't contain a
> P2P bridge that would require probing further below the FPGA
> itself.
>
> The basic idea is to call pci_scan_slot() on the devfn where
> the FPGA is supposed to respond.
>
> Then you need to also do some fixup. First you need to call
> pcibios_setup_bus_devices(). This will wire up the device
> to an OF node if you have one, setup some default DMA ops,
> etc...
>
> Note that this function will walk over all devices on that bus
> which is interesting since some of those may have already been
> fully setup initially. Hopefully that isn't a problem. If it
> was to become one, we would have to figure out a way to skip
> devices that have already been "setup".
>
> And finally you call pcibios_finish_adding_to_bus() which will
> do the resource allocation pass on all new devices on the bus
> and register them with the core device layer.
>
> Cheers,
> Ben.
>
>   
Thanks for the advice. This approach worked well for me, with some minor
modifications. First I had to clear PPC_INDIRECT_TYPE_NO_PCIE_LINK
in hose->indirect_type, otherwise reads from configuration space fail. And
second, I had to call pci_assign_resource() to complete the assignment 
of FPGA
memory. The memory range is 0x0 - 0x7fffff, and PCI code doesn't like 
resources
starting at 0.

Again, thanks a lot for your help, much appreciated.

Felix.