PCIe/PCI rescan for new FPGA devices

PCIe/PCI rescan for new FPGA devices

[Xianghua Xiao]

I have a FPGA-PCIe device and a FPGA-PCI device that are empty at
kernel boot time, they're only loaded after kernel is up. After I load
FPGA images, "lspci" can not show the device, "echo 1 >
/sys/bus/pci/rescan" also does nothing about it. With a reboot I can
see the devices are allocated correctly though, but I want to avoid
the reboot.

I checked at fakephp(which is to be deprecated) and hotplug/rescan,
none of that worked for this scenario.

Can 'rescan' handle cases like this? or, is there a way that I tell
the kernel to reserve a few BARs somehow in the PCI topology at
bootime so I can "insert" the FPGAs later?

Thanks,
Xianghua

Re: PCIe/PCI rescan for new FPGA devices

[Greg KH]

On Thu, Apr 28, 2011 at 06:06:50PM -0500, Xianghua Xiao wrote:
> I have a FPGA-PCIe device and a FPGA-PCI device that are empty at
> kernel boot time, they're only loaded after kernel is up. After I load
> FPGA images, "lspci" can not show the device, "echo 1 >
> /sys/bus/pci/rescan" also does nothing about it. With a reboot I can
> see the devices are allocated correctly though, but I want to avoid
> the reboot.
> 
> I checked at fakephp(which is to be deprecated) and hotplug/rescan,
> none of that worked for this scenario.
> 
> Can 'rescan' handle cases like this? or, is there a way that I tell
> the kernel to reserve a few BARs somehow in the PCI topology at
> bootime so I can "insert" the FPGAs later?

Your firmware has to do this, see the PCI hotplug specification for
details as to the proper procedure for this.

Best of luck,

greg k-h

Re: PCIe/PCI rescan for new FPGA devices

[Yinghai Lu]

On Thu, Apr 28, 2011 at 4:06 PM, Xianghua Xiao <xiaoxianghua@gmail.com> wrote:
> I have a FPGA-PCIe device and a FPGA-PCI device that are empty at
> kernel boot time, they're only loaded after kernel is up. After I load
> FPGA images, "lspci" can not show the device, "echo 1 >
> /sys/bus/pci/rescan" also does nothing about it. With a reboot I can
> see the devices are allocated correctly though, but I want to avoid
> the reboot.
>
> I checked at fakephp(which is to be deprecated) and hotplug/rescan,
> none of that worked for this scenario.
>
> Can 'rescan' handle cases like this? or, is there a way that I tell
> the kernel to reserve a few BARs somehow in the PCI topology at
> bootime so I can "insert" the FPGAs later?

well, you did not send out the lspci -tv yet.

assume lspci tree like this

 +-[0000:c0]-+-00.0-
 |           |
 |           +-01.0-[c3]--
 |           +-03.0-[c4-c9]----00.0-[c5-c7]--+-02.0-[c6]--+-00.0 your FPGA

so c0:03.0 is pcie root port,
c4:00.0 is your pcie switch upstream port
c5:02.0 is your pcie switch downstream port
c6:0 is your FPGA controller.

so you could remove c0:03.0 or c4:00.0 or c5:02.0, then do rescan.
--- depend if your BIOS allocate the right value to the c0:03.0

Yinghai