Re: XFS NVMe RDMA?

[Christoph Hellwig <hch@infradead.org>]

Hi Dan,

sorry for the late reply.  This sat in my outbox for a while almost
fully written.

On Wed, Oct 27, 2021 at 09:23:42AM +0100, Dan Greenfield wrote:
> Quick Question: would I be correct in assuming that if I mmap()ed each 8-64MB (chunk) file from XFS, and then did RDMA from the mmap region, that it would first be copied from NVMe into DRAM (does this bypass CPU?) and *then* be copied across RDMA, rather than directly be copied from NVMe by RDMA? Or does O_DIRECT properly allow bypass straight to NVMe for RDMA?

The answer is: it depends.  mmap on a non-DAX file system always copied
into DRAM.  mmap on a DAX file system (that is using pmem) can map
the "storage" directly into memory, in which case some RDMA setups can
DMA without a copy.

But with a plain old SSD there is no path available to userspace to
transfer without copying to DRAM.  If OTOH you use in-kernel NVMe over
fabics target, it can directly transfers from the SSDs in some
circumstances.  Currently that does require using the SSD directly
without a file system, but with a little more work it could also work
using a file system.