Convention for PCI device to handle bridge window change

Convention for PCI device to handle bridge window change

[Shelton, Ben]

Hi all,

I'm working on an issue with an x64-based emulation system that has
multiple PCI-accessible FPGAs, each of which sits behind a PCI bridge.

We program the first FPGA and then warm-reboot the CPU.  When it comes
back up, there is an additional PCIe device that has been programmed
into the FPGA, and the kernel goes through the remapping process to
figure out how to fit everything.  The remapping succeeds by moving
several of the FPGA bridges out of the way.  The bridges get set up
properly, but the FPGA resources themselves get released but not
reassigned, and the BAR of the FPGA behind each bridge that got
relocated is still set to the old address.  If I add a hack in the boot
process to call pci_assign_resource() for those particular devices if
they have not been assigned, then everything works as expected.

Where is the usual place to handle updating the BAR for a device after
the bridge window of the bridge it's behind has changed?

Is it in the driver?  Looking at other drivers, there are a few that
call pci_assign_resource() explicitly if their resource has not been
assigned, e.g. drivers/char/agp/intel-agp.c.  There are also a few
drivers that used to do this but no longer do, e.g. ath10k (see commit
a6a2f74e for where it got removed).  

Or are resources behind a bridge that was relocated supposed to be
reassigned as part of the remapping process, and that just isn't
happening for some reason in our case?

Thanks,
Ben

Re: Convention for PCI device to handle bridge window change

[Bjorn Helgaas]

Hi Ben,

Sorry for the late response.  I didn't notice this until now.

On Thu, Aug 20, 2015 at 12:00:00PM -0500, Shelton, Ben wrote:
> Hi all,
> 
> I'm working on an issue with an x64-based emulation system that has
> multiple PCI-accessible FPGAs, each of which sits behind a PCI bridge.
> 
> We program the first FPGA and then warm-reboot the CPU.  When it comes
> back up, there is an additional PCIe device that has been programmed
> into the FPGA, and the kernel goes through the remapping process to
> figure out how to fit everything.  The remapping succeeds by moving
> several of the FPGA bridges out of the way.  The bridges get set up
> properly, but the FPGA resources themselves get released but not
> reassigned, and the BAR of the FPGA behind each bridge that got
> relocated is still set to the old address.  If I add a hack in the boot
> process to call pci_assign_resource() for those particular devices if
> they have not been assigned, then everything works as expected.

When you say "warm-reboot the CPU," I assume you mean that you're
rebooting Linux from scratch, and it has no knowledge about the
previous boot.  But apparently BIOS isn't involved?  At least on x86,
BIOS usually assigns space for most or all PCI devices, but in your
case it sounds like Linux is doing the assignment.

> Where is the usual place to handle updating the BAR for a device after
> the bridge window of the bridge it's behind has changed?

If the Linux PCI core is changing a bridge window, it should also
change the BARs of any devices below the bridge.  Maybe a complete
dmesg log would help me understand what's happening.

Bjorn