Re: Sharing PCIe MMIO with other Drivers

[Oliver O'Halloran <oohall@gmail.com>]

On Fri, 2018-11-09 at 16:22 -0800, Andrei Danaila wrote:
> Hello,
> 
> I have a question about best practices in writing an PCIe driver for an FPGA. If this is not the best place to ask, please let me know.
> 
> I have an FPGA which is connected over PCIe to an x86 host. The FPGA has a variety of peripherals on it, I2C, UART, SPI etc. All of these peripherals can be accessed from the host by accessing different offsets from the BAR0 address.
> 
> I am running linux kernel 4.14 on the host and have written a PCIe device driver which probes off the device id manufacturer ID of the FPGA.
> 
> The device driver calls pci_iomap( to obtain the cookie used to access the BAR. This works fine and via this mechanism I can read/write to the FPGA address space after calling ioremap on the cookie.
> 
> What I am trying to do now however is create a I2C platform device representing the I2C bus on the FPGA and add to it, as a resource, the BAR0 address + the I2C offset, to get the host's i2c driver to probe off this new PCI device.

Keep in mind the address you read out of BAR0 in config space is a PCI
bus address which isn't necessarily usable as a system physical
address. It might be fine on x86 though.

> 
> In addition I am also trying to add an IRQ number for the I2C driver to use which is an MSIX mapped interrupt number obtained via pci_irq_vector.

You should be able to do something with an irqchip to forward the
interrupt to the platform device safely.

> 
> In essence, I am trying to get the x86 host to own this device exposed via io-remapped region in PCI land, and use its driver to manage it.
> 
> The problem I am having is that I am getting a EBUSY return code when I try to register the resource to the platform device, after the pci_iomap has taken place.
> 
> The resource type is IORESOURCE_SYSTEM_RAM | IORESOURCE_MUXED and the start of the resource is the BAR0 address as returned by pci_iomap + I2C_OFFSET.

IIRC IORESOURCE_SYSTEM_RAM marks the resource as being normal system
RAM (i.e. cacheable) rather than MMIO space, so this probably not the
right set of flags to use.

Anyway, trying to create resource from the result of pci_iomap() sounds
a little fishy to me since memory resources generally contain physical
addresses and pci_iomap() should be returning a virtual address. The
-EBUSY is probably due to the resource you created conflicting with an
existing resource that doesn't have the MUXED flag set. I don't
remember the exact process for doing this properly, but If I were you I
would:

a) convert your resource to use physical addresses
b) make the BAR resource the parent of your new resource.

and see if that fixes the problem.

Oliver