Re: [RFC PATCH v2 0/3] RDMA: add dma-buf support

[Jason Gunthorpe <jgg@ziepe.ca>]

On Mon, Jun 29, 2020 at 10:31:40AM -0700, Jianxin Xiong wrote:

> ZONE_DEVICE is a new zone for device memory in the memory management
> subsystem. It allows pages from device memory being described with
> specialized page structures. As the result, calls like get_user_pages()
> can succeed, but what can be done with these page structures may be

get_user_pages() does not succeed with ZONE_DEVICE_PAGEs

> Heterogeneous Memory Management (HMM) utilizes mmu_interval_notifier
> and ZONE_DEVICE to support shared virtual address space and page
> migration between system memory and device memory. HMM doesn't support
> pinning device memory because pages located on device must be able to
> migrate to system memory when accessed by CPU. Peer-to-peer access
> is possible if the peer can handle page fault. For RDMA, that means
> the NIC must support on-demand paging.

peer-peer access is currently not possible with hmm_range_fault().

> This patch series adds dma-buf importer role to the RDMA driver in
> attempt to support RDMA using device memory such as GPU VRAM. Dma-buf is
> chosen for a few reasons: first, the API is relatively simple and allows
> a lot of flexibility in implementing the buffer manipulation ops.
> Second, it doesn't require page structure. Third, dma-buf is already
> supported in many GPU drivers. However, we are aware that existing GPU
> drivers don't allow pinning device memory via the dma-buf interface.

So.. this patch doesn't really do anything new? We could just make a
MR against the DMA buf mmap and get to the same place?

> Pinning and mapping a dma-buf would cause the backing storage to migrate
> to system RAM. This is due to the lack of knowledge about whether the
> importer can perform peer-to-peer access and the lack of resource limit
> control measure for GPU. For the first part, the latest dma-buf driver
> has a peer-to-peer flag for the importer, but the flag is currently tied
> to dynamic mapping support, which requires on-demand paging support from
> the NIC to work.

ODP for DMA buf?

> There are a few possible ways to address these issues, such as
> decoupling peer-to-peer flag from dynamic mapping, allowing more
> leeway for individual drivers to make the pinning decision and
> adding GPU resource limit control via cgroup. We would like to get
> comments on this patch series with the assumption that device memory
> pinning via dma-buf is supported by some GPU drivers, and at the
> same time welcome open discussions on how to address the
> aforementioned issues as well as GPU-NIC peer-to-peer access
> solutions in general.

These seem like DMA buf problems, not RDMA problems, why are you
asking these questions with a RDMA patch set? The usual DMA buf people
are not even Cc'd here.

> This is the second version of the patch series. Here are the changes
> from the previous version:
> * Instead of adding new device method for dma-buf specific registration,
> existing method is extended to accept an extra parameter.

I think the comment was the extra parameter should have been a umem or
maybe a new umem_description struct, not blindly adding a fd as a
parameter and a wack of EOPNOTSUPPS

> This series is organized as follows. The first patch adds the common
> code for importing dma-buf from a file descriptor and pinning and
> mapping the dma-buf pages. Patch 2 extends the reg_user_mr() method
> of the ib_device structure to accept dma-buf file descriptor as an extra
> parameter. Vendor drivers are updated with the change. Patch 3 adds a
> new uverbs command for registering dma-buf based memory region.

The ioctl stuff seems OK, but this doesn't seem to bring any new
functionality?

Jason