Re: [RFC PATCH 2/4] nfs: add NFS_CAP_P2PDMA and detect transport support

["Chuck Lever" <cel@kernel.org>]

On Tue, Apr 14, 2026, at 12:54 PM, Pranjal Shrivastava wrote:
> On Thu, Apr 02, 2026 at 09:11:04AM -0400, Chuck Lever wrote:
>> 
>> On Wed, Apr 1, 2026, at 3:44 PM, Pranjal Shrivastava wrote:
>> > The NFS server capabilities bitmask (server->caps) is currently full,
>> > utilizing all 32 bits of the existing unsigned int. Expand the bitmask
>> > to 64 bits (u64) to allow for new feature flags.
>> >
>> > Introduce a new capability bit, NFS_CAP_P2PDMA, to indicate that the
>> > local mount is backed by hardware and a transport capable of PCI
>> > Peer-to-Peer DMA.
>> >
>> > Update nfs_server_set_init_caps() to query the underlying SunRPC
>> > transport for P2PDMA support during the mount process. If the transport
>> > (e.g., RDMA) signals support, set the NFS_CAP_P2PDMA bit in the mount's
>> > capabilities. This allows the high-performance Direct I/O path to
>> > efficiently determine if it should allow P2P memory buffers.
>> 
>> > diff --git a/fs/nfs/client.c b/fs/nfs/client.c
>> > index be02bb227741..f177cf098d44 100644
>> > --- a/fs/nfs/client.c
>> > +++ b/fs/nfs/client.c
>> 
>> > @@ -725,6 +727,12 @@ void nfs_server_set_init_caps(struct nfs_server *server)
>> >  		nfs4_server_set_init_caps(server);
>> >  		break;
>> >  	}
>> > +
>> > +	rcu_read_lock();
>> > +	xprt = rcu_dereference(server->client->cl_xprt);
>> > +	if (xprt->ops->supports_p2pdma && xprt->ops->supports_p2pdma(xprt))
>> > +		server->caps |= NFS_CAP_P2PDMA;
>> > +	rcu_read_unlock();
>> >  }
>> >  EXPORT_SYMBOL_GPL(nfs_server_set_init_caps);
>> 
>> Is the transport even connected when the NFS client does this
>> test? If it isn't, xprtrdma and the RDMA core have not chosen
>> an underlying device yet.
>> 
>> Note that, even if this logic /is/ correct, if the transport
>> connection is lost the transport will reconnect automatically,
>> doing the RDMA CM dance again and possibly resolving to a
>> different device. The NFS client layer will be none-the-wiser
>> and the NFS_CAP_P2PDMA flag setting will be stale at that point,
>> and quite possibly incorrect if the new connection's device is
>> not P2P-enabled.
>> 
>> (Basically this is what happens when an RDMA device is removed).
>> 
>> So this detection has to be done as part of xprtrdma's connection
>> flow, and it needs to set a flag somewhere in the rpc_xprt. The
>> NFS direct I/O code path then has to look for that flag before
>> choosing the mechanism/flags it uses for each iov iter.
>> 
>
> Ack. I agree, so should we start with an inital cap and then update it 
> in the event of a transport change / disconnect? Or shall we populate 
> the cap only when a transport is connected?

IMO this flag does not belong in the NFS server CAPS, as it is a
capability associated with each RPC transport. How should
NFS_CAP_P2PDMA be set if there are two RPC transports, one with
P2PDMA enabled and with it disabled? (Perhaps it should be a flag
in the transport switch instance rather than the transport instance).

Which mechanism to use has to be re-decided every time a dreq is
scheduled because the xprt can change between an original send
and a retransmission (if, say, the COMMIT verifier changes due to
a server reboot).

Trond and Anna will have the final say about how this works.


-- 
Chuck Lever