[RFC PATCH 0/4] nfs: Enable PCI Peer-to-Peer DMA (P2PDMA) support

Pranjal Shrivastava
2026-04-01T19:45:07Z

As high-performance storage environments increasingly rely on direct
data movement between PCIe endpoints (e.g., moving data directly between
an NVMe Controller Memory Buffer and a Network Interface), support for
Peer-to-Peer DMA (P2PDMA) in the network filesystem layer becomes
essential. This series introduces P2PDMA support for the NFS Direct I/O.

Currently, NFS O_DIRECT operations fail with -EREMOTEIO if the user
buffer resides in PCIe BAR memory. This is primarily due to the use of
the legacy `iov_iter_get_pages_alloc2()` API, which cannot pass the
required `FOLL_PCI_P2PDMA` flag, and a request lifecycle that is unaware
of the pinning requirements for P2P memory.

Design
=======
The proposed design centers around making the NFS request lifecycle
"pin-aware" and upgrading the infrastructure to support modern memory
extraction APIs.

1. 64-bit Capability Infrastructure
The existing nfs_server->caps bitmask is limited to 32 bits and is
currently exhausted. This series expands the bitmask to 64 bits to
accommodate NFS_CAP_P2PDMA. Crucially, it also refactors the NFS_CAP_*
constants to use ULL definitions. This prevents a subtle 32-bit
truncation bug where bitwise negations (e.g., caps &= ~NFS_CAP_ACLS)
would accidentally clear the high bits of the 64-bit capability field.

2. Transport-Level Detection
P2PDMA support is a property of the local transport hardware. A new
supports_p2pdma operation is added to the SunRPC transport ops. For RDMA,
this is implemented by querying the underlying device via
ib_dma_pci_p2p_dma_supported(). The NFS client queries this during mount
and sets the NFS_CAP_P2PDMA bit accordingly.

3. Pin-Aware Request Lifecycle
Standard NFS requests use get_page() and put_page() for memory
management. However, memory extracted via iov_iter_extract_pages()
requires explicit pinning and unpinning (unpin_user_page()).

This series introduces a PG_PINNED flag in struct nfs_page. When set,
the request lifecycle skips standard page referencing and ensures that
unpin_user_page() is called only when the I/O is complete. This ensures
that physical memory remains pinned for the duration of the DMA transfer

4. API Migration
The Direct I/O path is migrated to the modern iov_iter_extract_pages()
API. The ITER_ALLOW_P2PDMA flag is passed to the iterator only when the
local mount has signaled P2P support via the capability bit. This ensures
that "normal" users on standard TCP/UDP transports see no change in
behavior or overhead.

Call for review
===============
Any insights on the proposed changes to the nfs_page lifecycle and the
64-bit capability expansion are appreciated. If this approach is deemed
incorrect or if there is a more idiomatic way for this, please direct me
in the right direction.

Thanks,
Praan

Pranjal Shrivastava (4):
sunrpc: add supports_p2pdma to rpc_xprt_ops
nfs: add NFS_CAP_P2PDMA and detect transport support
nfs: make nfs_page pin-aware
nfs: allow P2PDMA in direct I/O path

fs/nfs/client.c | 8 ++++
fs/nfs/direct.c | 51 ++++++++++++++++++-------
fs/nfs/nfs4_fs.h | 2 +-
fs/nfs/pagelist.c | 18 ++++++---
fs/nfs/super.c | 2 +-
include/linux/nfs_fs_sb.h | 67 +++++++++++++++++----------------
include/linux/nfs_page.h | 2 +
include/linux/sunrpc/xprt.h | 1 +
net/sunrpc/xprtrdma/transport.c | 9 +++++
9 files changed, 106 insertions(+), 54 deletions(-)

--
2.53.0.1185.g05d4b7b318-goog