Re: [PATCH] Make RPCRDMA_MAX_RECV_BATCH configurable.

[Chuck Lever <chuck.lever@oracle.com>]

On 11/13/25 11:39 AM, gaurav gangalwar wrote:
> On Thu, Nov 13, 2025 at 7:49 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>>
>> On 11/13/25 4:37 AM, Gaurav Gangalwar wrote:
>>> Bumped up rpcrdma_max_recv_batch to 64.
>>> Added param to change to it, it becomes handy to use higher value
>>> to avoid hung.
>>
>> [ Resend with correct NFSD reviewer email addresses and linux-rdma@ ]
>>
>> Hi Gaurav -
>>
>> Adding an administrative setting is generally a last resort. First,
>> we want a full root-cause analysis to understand the symptoms you
>> are trying to address. Do you have an RCA or a simple reproducer to
>> share with us?
> 
> Issue found while testing fio workload over RDMA
> Client: Ubuntu 24.04
> Server: Ganesha NFS server
> We have seen intermittent hung on client with buffered IO workload at
> large scale with around 30 RDMA connections, client was under memory
> pressure.
> Ganesha log shows
> 
> 10/11/2025 16:39:12Z : ntnx-10-57-210-224-a-fsvm 1309416[none]
> [0x7f49a6c3fe80] rpc :TIRPC :EVENT :rpc_rdma_cq_event_handler() cq
> completion status: RNR retry counter exceeded (13) rdma_xprt state 5
> opcode 2 cbc 0x7f4996688000 inline 1
> 
> Which points to lack of posted recv buffers on client.
> Once we increased rpcrdma_max_recv_batch to 64, issue was resolved.

That still doesn't convince me that increasing the receive batch count
is a good fix, though it's certainly a workaround.

The client's RPC/RDMA code is supposed to track the number of Sends and
keep the correct number of Receives on the Receive Queue. The goal of
the implementation is to never encounter an RNR.

Therefore, if it's not doing that (and the RNR retries suggests that's
the case) there is an actual bug somewhere. The extra batch Receives are
an optimization, and should have no impact on correct operation.

If you can't reproduce this with the Linux NFS server, the place to
start looking for misbehavior is NFS/Ganesha, as it is the newer NFS
over RDMA implementation of the two servers. Maybe it's not handling
credit accounting correctly, or perhaps it's putting more Sends on
the wire than the credit limit allows.


-- 
Chuck Lever