Re: [PATCH 11/14] PCI/P2PDMA: dma_map P2PDMA map requests that traverse the host bridge

Jason Gunthorpe
2019-07-25T16:34:52Z

On Thu, Jul 25, 2019 at 10:00:25AM -0600, Logan Gunthorpe wrote:
>
>
> On 2019-07-25 12:10 a.m., Christoph Hellwig wrote:
> > On Wed, Jul 24, 2019 at 09:58:59AM -0600, Logan Gunthorpe wrote:
> >>
> >>
> >> On 2019-07-24 12:32 a.m., Christoph Hellwig wrote:
> >>>> struct dev_pagemap *pgmap = sg_page(sg)->pgmap;
> >>>> + struct pci_dev *client;
> >>>> + int dist;
> >>>> +
> >>>> + client = find_parent_pci_dev(dev);
> >>>> + if (WARN_ON_ONCE(!client))
> >>>> + return 0;
> >>>>
> >>>> + dist = upstream_bridge_distance(pgmap->pci_p2pdma_provider,
> >>>> + client, NULL);
> >>>
> >>> Doing this on every mapping call sounds expensive..
> >>
> >> The result of this function is cached in an xarray (per patch 4) so, on
> >> the hot path, it should just be a single xa_load() which should be a
> >> relatively fast lookup which is similarly used for other hot path
> >> operations.
> >
> > We don't cache find_parent_pci_dev, though. So we should probably
> > export find_parent_pci_dev with a proper namespaces name and cache
> > that in the caler.
>
> Oh, yes, I'll take a look at this. Of the two callers: NVMe should be
> easy we could just pass the PCI device instead of the struct device.
> RDMA is significantly more unclear: would you add a pci_dev to struct
> ib_device? Or maybe we should be able to simply rely on the fact that
> the DMA device *must* be a PCI device and just use to_pci_dev() directly?

AFAIK you need to use the ib_device->dma_device and add some kind of
is_pci_dev to make it safe

Jason