Re: [PATCH rdma-next] RDMA: Explicitly pass in the dma_device to ib_register_device

[Jason Gunthorpe <jgg@nvidia.com>]

On Thu, Sep 24, 2020 at 07:13:25AM +0100, Christoph Hellwig wrote:
> On Thu, Sep 24, 2020 at 06:53:45AM +0100, Christoph Hellwig wrote:
> > Because it isn't DMA, and not we need crappy workarounds like the one
> > in the PCIe P2P code.  It also enforces the same size for dma_addr_t
> > and a pointer, which is not true in various cases.  More importantly
> > it forces a very strange design in the IB APIs (actually it seems the
> > other way around - the weird IB Verbs APIs forced this decision, but
> > it cements it in).  For example most modern Mellanox cards can include
> > immediate data in the command capsule (sorry NVMe terms, I'm pretty
> > sure you guys use something else) that is just copied into the BAR
> > using MMIO.  But the IB API design still forces the ULP to dma map
> > it, which is idiotic.
> 
> FYI, here is the quick patch to kill of dma_virt_ops on top of the
> patch under discussion here.  I think the diffstat alone speaks for
> itself, never mind the fact that we avoid indirect calls for the
> drivers that don't want the DMA mapping.

Yes looks reasonable, I can't recall why things went in the other
direction.

> diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
> index bf66b0622b49c8..f2621eaecb7211 100644
> +++ b/drivers/pci/p2pdma.c
> @@ -556,15 +556,6 @@ int pci_p2pdma_distance_many(struct pci_dev *provider, struct device **clients,
>  		return -1;
>  
>  	for (i = 0; i < num_clients; i++) {
> -#ifdef CONFIG_DMA_VIRT_OPS
> -		if (clients[i]->dma_ops == &dma_virt_ops) {
> -			if (verbose)
> -				dev_warn(clients[i],
> -					 "cannot be used for peer-to-peer DMA because the driver makes use of dma_virt_ops\n");
> -			return -1;
> -		}
> -#endif

We still need to block p2p for any of the non DMA devices. We can't
pass a p2p sgl into SW devices, they will try to memcpy from the MMIO
__iommem which is not allowed.

Now the check changes to !ib_device->dma_device in the callers

Jason