Re: [PATCH v6 0/6] Improve PCI memory mapping API

[Bjorn Helgaas <helgaas@kernel.org>]

On Tue, Oct 22, 2024 at 10:51:53AM +0900, Damien Le Moal wrote:
> On 10/22/24 07:19, Bjorn Helgaas wrote:
> > On Sat, Oct 12, 2024 at 08:32:40PM +0900, Damien Le Moal wrote:
> >> This series introduces the new functions pci_epc_mem_map() and
> >> pci_epc_mem_unmap() to improve handling of the PCI address mapping
> >> alignment constraints of endpoint controllers in a controller
> >> independent manner.
> > 
> > Hi Damien, I know this is obvious to everybody except me, but who
> > uses this mapping?  A driver running on the endpoint that does
> > MMIO?  DMA (Memory Reads or Writes that target an endpoint BAR)?
> > I'm still trying to wrap my head around the whole endpoint driver
> > model.
> 
> The mapping API is for mmio or DMA using endpoint controller memory
> mapped to a host PCI address range. It is not for BARs. BARs setup
> does not use the same API and has not changed with these patches.
> 
> BARs can still be accessed on the endpoint (within the EPF driver)
> with regular READ_ONCE()/WRITE_ONCE() once they are set. But any
> data transfer between the PCI RC host and the EPF driver on the EP
> host that use mmio or DMA generic channel (memory copy offload
> channel) needs the new mapping API. DMA transfers that can be done
> using dedicated DMA rx/tx channels associated with the endpoint
> controller do not need to use this API as the mapping to the host
> PCI address space is automatically handled by the DMA driver.

Sorry I'm dense.  I'm really not used to thinking in the endpoint
point of view.  Correct me where I go off the rails:

  - This code (pci_epc_mem_map()) runs on an endpoint, basically as
    part of some endpoint firmware, right?

  - On the endpoint's PCIe side, it receives memory read/write TLPs?

  - These TLPs would be generated elsewhere in the PCIe fabric, e.g.,
    by a driver on the host doing MMIO to the endpoint, or possibly
    another endpoint doing peer-to-peer DMA.

  - Mem read/write TLPs are routed by address, and the endpoint
    accepts them if the address matches one of its BARs.

  - This is a little different from a Root Complex, which would
    basically treat reads/writes to anything outside the Root Port
    windows as incoming DMA headed to physical memory.

  - A Root Complex would use the TLP address (the PCI bus address)
    directly as a CPU physical memory address unless the TLP address
    is translated by an IOMMU.

  - For the endpoint, you want the BAR to be an aperture to physical
    memory in the address space of the SoC driving the endpoint.

  - The SoC physical memory address may be different from the PCI but
    address in the TLP, and pci_epc_mem_map() is the way to account
    for this?

  - IOMMU translations from PCI to CPU physical address space are
    pretty arbitrary and needn't be contiguous on the CPU side.

  - pci_epc_mem_map() sets up a conceptually similar PCI to CPU
    address space translation, but it's much simpler because it
    basically applies just a constant offset?