Re: [RFC PATCH 2/4] nfs: add NFS_CAP_P2PDMA and detect transport support

Chuck Lever
2026-04-14T20:59:34Z

On Tue, Apr 14, 2026, at 12:54 PM, Pranjal Shrivastava wrote:
> On Thu, Apr 02, 2026 at 09:11:04AM -0400, Chuck Lever wrote:
>>
>> On Wed, Apr 1, 2026, at 3:44 PM, Pranjal Shrivastava wrote:
>> > The NFS server capabilities bitmask (server->caps) is currently full,
>> > utilizing all 32 bits of the existing unsigned int. Expand the bitmask
>> > to 64 bits (u64) to allow for new feature flags.
>> >
>> > Introduce a new capability bit, NFS_CAP_P2PDMA, to indicate that the
>> > local mount is backed by hardware and a transport capable of PCI
>> > Peer-to-Peer DMA.
>> >
>> > Update nfs_server_set_init_caps() to query the underlying SunRPC
>> > transport for P2PDMA support during the mount process. If the transport
>> > (e.g., RDMA) signals support, set the NFS_CAP_P2PDMA bit in the mount's
>> > capabilities. This allows the high-performance Direct I/O path to
>> > efficiently determine if it should allow P2P memory buffers.
>>
>> > diff --git a/fs/nfs/client.c b/fs/nfs/client.c
>> > index be02bb227741..f177cf098d44 100644
>> > --- a/fs/nfs/client.c
>> > +++ b/fs/nfs/client.c
>>
>> > @@ -725,6 +727,12 @@ void nfs_server_set_init_caps(struct nfs_server *server)
>> > nfs4_server_set_init_caps(server);
>> > break;
>> > }
>> > +
>> > + rcu_read_lock();
>> > + xprt = rcu_dereference(server->client->cl_xprt);
>> > + if (xprt->ops->supports_p2pdma && xprt->ops->supports_p2pdma(xprt))
>> > + server->caps |= NFS_CAP_P2PDMA;
>> > + rcu_read_unlock();
>> > }
>> > EXPORT_SYMBOL_GPL(nfs_server_set_init_caps);
>>
>> Is the transport even connected when the NFS client does this
>> test? If it isn't, xprtrdma and the RDMA core have not chosen
>> an underlying device yet.
>>
>> Note that, even if this logic /is/ correct, if the transport
>> connection is lost the transport will reconnect automatically,
>> doing the RDMA CM dance again and possibly resolving to a
>> different device. The NFS client layer will be none-the-wiser
>> and the NFS_CAP_P2PDMA flag setting will be stale at that point,
>> and quite possibly incorrect if the new connection's device is
>> not P2P-enabled.
>>
>> (Basically this is what happens when an RDMA device is removed).
>>
>> So this detection has to be done as part of xprtrdma's connection
>> flow, and it needs to set a flag somewhere in the rpc_xprt. The
>> NFS direct I/O code path then has to look for that flag before
>> choosing the mechanism/flags it uses for each iov iter.
>>
>
> Ack. I agree, so should we start with an inital cap and then update it
> in the event of a transport change / disconnect? Or shall we populate
> the cap only when a transport is connected?

IMO this flag does not belong in the NFS server CAPS, as it is a
capability associated with each RPC transport. How should
NFS_CAP_P2PDMA be set if there are two RPC transports, one with
P2PDMA enabled and with it disabled? (Perhaps it should be a flag
in the transport switch instance rather than the transport instance).

Which mechanism to use has to be re-decided every time a dreq is
scheduled because the xprt can change between an original send
and a retransmission (if, say, the COMMIT verifier changes due to
a server reboot).

Trond and Anna will have the final say about how this works.

--
Chuck Lever