Sharing PCIe MMIO with other Drivers

Sharing PCIe MMIO with other Drivers

[Andrei Danaila]

Hello,

I have a question about best practices in writing an PCIe driver for an FPGA. If this is not the best place to ask, please let me know.

I have an FPGA which is connected over PCIe to an x86 host. The FPGA has a variety of peripherals on it, I2C, UART, SPI etc. All of these peripherals can be accessed from the host by accessing different offsets from the BAR0 address.

I am running linux kernel 4.14 on the host and have written a PCIe device driver which probes off the device id manufacturer ID of the FPGA.

The device driver calls pci_iomap( to obtain the cookie used to access the BAR. This works fine and via this mechanism I can read/write to the FPGA address space after calling ioremap on the cookie.

What I am trying to do now however is create a I2C platform device representing the I2C bus on the FPGA and add to it, as a resource, the BAR0 address + the I2C offset, to get the host's i2c driver to probe off this new PCI device.

In addition I am also trying to add an IRQ number for the I2C driver to use which is an MSIX mapped interrupt number obtained via pci_irq_vector.

In essence, I am trying to get the x86 host to own this device exposed via io-remapped region in PCI land, and use its driver to manage it.

The problem I am having is that I am getting a EBUSY return code when I try to register the resource to the platform device, after the pci_iomap has taken place.

The resource type is IORESOURCE_SYSTEM_RAM | IORESOURCE_MUXED and the start of the resource is the BAR0 address as returned by pci_iomap + I2C_OFFSET.

In the I2C device driver, I am expecting to do an ioremap on the resource and be able to access it by de-refencing


A couple of questions:

1. Is this the correct software flow for managing multiple devices exposed by a PCIe BAR0 address space?
    If not, what is the correct flow?
    If yes, any ideas on what may be going wrong?

Please feel free to point me to any examples, I have looked around quite a bit but did not manage to find enough detail to let me solve this problem.

Thank you

Re: Sharing PCIe MMIO with other Drivers

[Sinan Kaya]

On 11/9/2018 4:22 PM, Andrei Danaila wrote:
> A couple of questions:
> 
> 1. Is this the correct software flow for managing multiple devices exposed by a PCIe BAR0 address space?
>      If not, what is the correct flow?
>      If yes, any ideas on what may be going wrong?

General practice is to create a PCI Physical Function for each sub-functionality
(I2C/UART/DMA etc.) so that individual drivers can claim their own PCI device
and its own BAR space without sharing resources.

Re: Sharing PCIe MMIO with other Drivers

[Andrei Danaila]

Thanks Sinan,

I thought of that as well, unfortunately that it is not possible since the FPGA IP block is proprietary from a 3P vendor and lumps everything under one function with no option of adding additional functions.



On Fri, Nov 9, 2018, at 4:32 PM, Sinan Kaya wrote:
> On 11/9/2018 4:22 PM, Andrei Danaila wrote:
> > A couple of questions:
> > 
> > 1. Is this the correct software flow for managing multiple devices exposed by a PCIe BAR0 address space?
> >      If not, what is the correct flow?
> >      If yes, any ideas on what may be going wrong?
> 
> General practice is to create a PCI Physical Function for each sub-functionality
> (I2C/UART/DMA etc.) so that individual drivers can claim their own PCI device
> and its own BAR space without sharing resources.

Re: Sharing PCIe MMIO with other Drivers

[Sinan Kaya]

On 11/9/2018 4:43 PM, Andrei Danaila wrote:
> Thanks Sinan,
> 
> I thought of that as well, unfortunately that it is not possible since the FPGA IP block is proprietary from a 3P vendor and lumps everything under one function with no option of adding additional functions.

How about adding multiple BARs and then give each BAR to a children driver?

Re: Sharing PCIe MMIO with other Drivers

[Andrei Danaila]

Unfortunately, I cannot change the IP on the FPGA to add more BARs either, the whole thing is locked down.

On Fri, Nov 9, 2018, at 4:48 PM, Sinan Kaya wrote:
> On 11/9/2018 4:43 PM, Andrei Danaila wrote:
> > Thanks Sinan,
> > 
> > I thought of that as well, unfortunately that it is not possible since the FPGA IP block is proprietary from a 3P vendor and lumps everything under one function with no option of adding additional functions.
> 
> How about adding multiple BARs and then give each BAR to a children driver?

Re: Sharing PCIe MMIO with other Drivers

[Sinan Kaya]

On 11/9/2018 4:57 PM, Andrei Danaila wrote:
> Unfortunately, I cannot change the IP on the FPGA to add more BARs either, the whole thing is locked down.

Not the best hardware.
OK. I am running out of ideas. It doesn't play nice with the driver development
model.

One idea is to have a parent PCI driver to allocate BAR space and have children
platform devices that you create at runtime. You can point to arbitrary address
in the system memory map and an IRQ using the platform devices. Though, it might
have some alignment requirement in order to be able to make MMU mappings.

Re: Sharing PCIe MMIO with other Drivers

[Oliver O'Halloran]

On Fri, 2018-11-09 at 16:22 -0800, Andrei Danaila wrote:
> Hello,
> 
> I have a question about best practices in writing an PCIe driver for an FPGA. If this is not the best place to ask, please let me know.
> 
> I have an FPGA which is connected over PCIe to an x86 host. The FPGA has a variety of peripherals on it, I2C, UART, SPI etc. All of these peripherals can be accessed from the host by accessing different offsets from the BAR0 address.
> 
> I am running linux kernel 4.14 on the host and have written a PCIe device driver which probes off the device id manufacturer ID of the FPGA.
> 
> The device driver calls pci_iomap( to obtain the cookie used to access the BAR. This works fine and via this mechanism I can read/write to the FPGA address space after calling ioremap on the cookie.
> 
> What I am trying to do now however is create a I2C platform device representing the I2C bus on the FPGA and add to it, as a resource, the BAR0 address + the I2C offset, to get the host's i2c driver to probe off this new PCI device.

Keep in mind the address you read out of BAR0 in config space is a PCI
bus address which isn't necessarily usable as a system physical
address. It might be fine on x86 though.

> 
> In addition I am also trying to add an IRQ number for the I2C driver to use which is an MSIX mapped interrupt number obtained via pci_irq_vector.

You should be able to do something with an irqchip to forward the
interrupt to the platform device safely.

> 
> In essence, I am trying to get the x86 host to own this device exposed via io-remapped region in PCI land, and use its driver to manage it.
> 
> The problem I am having is that I am getting a EBUSY return code when I try to register the resource to the platform device, after the pci_iomap has taken place.
> 
> The resource type is IORESOURCE_SYSTEM_RAM | IORESOURCE_MUXED and the start of the resource is the BAR0 address as returned by pci_iomap + I2C_OFFSET.

IIRC IORESOURCE_SYSTEM_RAM marks the resource as being normal system
RAM (i.e. cacheable) rather than MMIO space, so this probably not the
right set of flags to use.

Anyway, trying to create resource from the result of pci_iomap() sounds
a little fishy to me since memory resources generally contain physical
addresses and pci_iomap() should be returning a virtual address. The
-EBUSY is probably due to the resource you created conflicting with an
existing resource that doesn't have the MUXED flag set. I don't
remember the exact process for doing this properly, but If I were you I
would:

a) convert your resource to use physical addresses
b) make the BAR resource the parent of your new resource.

and see if that fixes the problem.

Oliver