Re: [RFC PATCH 2/4] nfs: add NFS_CAP_P2PDMA and detect transport support

Pranjal Shrivastava
2026-04-14T19:55:09Z

On Thu, Apr 02, 2026 at 09:11:04AM -0400, Chuck Lever wrote:
>
> On Wed, Apr 1, 2026, at 3:44 PM, Pranjal Shrivastava wrote:
> > The NFS server capabilities bitmask (server->caps) is currently full,
> > utilizing all 32 bits of the existing unsigned int. Expand the bitmask
> > to 64 bits (u64) to allow for new feature flags.
> >
> > Introduce a new capability bit, NFS_CAP_P2PDMA, to indicate that the
> > local mount is backed by hardware and a transport capable of PCI
> > Peer-to-Peer DMA.
> >
> > Update nfs_server_set_init_caps() to query the underlying SunRPC
> > transport for P2PDMA support during the mount process. If the transport
> > (e.g., RDMA) signals support, set the NFS_CAP_P2PDMA bit in the mount's
> > capabilities. This allows the high-performance Direct I/O path to
> > efficiently determine if it should allow P2P memory buffers.
>
> > diff --git a/fs/nfs/client.c b/fs/nfs/client.c
> > index be02bb227741..f177cf098d44 100644
> > --- a/fs/nfs/client.c
> > +++ b/fs/nfs/client.c
>
> > @@ -725,6 +727,12 @@ void nfs_server_set_init_caps(struct nfs_server *server)
> > nfs4_server_set_init_caps(server);
> > break;
> > }
> > +
> > + rcu_read_lock();
> > + xprt = rcu_dereference(server->client->cl_xprt);
> > + if (xprt->ops->supports_p2pdma && xprt->ops->supports_p2pdma(xprt))
> > + server->caps |= NFS_CAP_P2PDMA;
> > + rcu_read_unlock();
> > }
> > EXPORT_SYMBOL_GPL(nfs_server_set_init_caps);
>
> Is the transport even connected when the NFS client does this
> test? If it isn't, xprtrdma and the RDMA core have not chosen
> an underlying device yet.
>
> Note that, even if this logic /is/ correct, if the transport
> connection is lost the transport will reconnect automatically,
> doing the RDMA CM dance again and possibly resolving to a
> different device. The NFS client layer will be none-the-wiser
> and the NFS_CAP_P2PDMA flag setting will be stale at that point,
> and quite possibly incorrect if the new connection's device is
> not P2P-enabled.
>
> (Basically this is what happens when an RDMA device is removed).
>
> So this detection has to be done as part of xprtrdma's connection
> flow, and it needs to set a flag somewhere in the rpc_xprt. The
> NFS direct I/O code path then has to look for that flag before
> choosing the mechanism/flags it uses for each iov iter.
>

Ack. I agree, so should we start with an inital cap and then update it
in the event of a transport change / disconnect? Or shall we populate
the cap only when a transport is connected?

Thanks,
Praan