Re: [PATCH] RDMA/rw: Fix MR pool exhaustion in bvec RDMA READ path

[Christoph Hellwig <hch@infradead.org>]

On Mon, Mar 09, 2026 at 11:46:21PM -0400, Chuck Lever wrote:
> Under NFS WRITE workloads the server performs RDMA READs to
> pull data from the client. With the inflated MR demand, the
> pool is rapidly exhausted, ib_mr_pool_get() returns NULL, and
> rdma_rw_init_one_mr() returns -EAGAIN. svcrdma treats this as
> a DMA mapping failure, closes the connection, and the client
> reconnects -- producing a cycle of 71% RPC retransmissions and
> ~100 reconnections per test run. RDMA WRITEs (NFS READ
> direction) are unaffected because DMA_TO_DEVICE never triggers
> the max_sgl_rd check.

So this changelog extensively describes the problem, but it doesn't
actually say how you fix it.

>  	/*
> +	 * IOVA not available; map each bvec individually. Do not
> +	 * check max_sgl_rd here: nr_bvec is a raw page count that
> +	 * overstates DMA entry demand and exhausts the MR pool.

It fails to explain why that is fine?

> +	 *
> +	 * TODO: A bulk DMA mapping API for bvecs analogous to
> +	 * dma_map_sgtable() would provide a proper post-DMA-
> +	 * coalescing segment count here, enabling the map_wrs
> +	 * path in more cases.

This isn't really something the DMA layer can easily do without getting
as inefficient as the sgtable based path.  What the block layer does
here is to simply keep a higher level count of merged segments.  The
other option would be to not create multiple bvecs for continguous
regions, which is what modern file system do in general, and why the
above block layer nr_phys_segments based optimization isn't actually
used all that much these days.

Why can't NFS send a single bvec for contiguous ranges?

> -	if (rdma_rw_io_needs_mr(dev, port_num, dir, nr_bvec))
> -		return rdma_rw_init_mr_wrs_bvec(ctx, qp, port_num, bvecs,
> -						nr_bvec, &iter, remote_addr,
> -						rkey, dir);
> -
>  	return rdma_rw_init_map_wrs_bvec(ctx, qp, bvecs, nr_bvec, &iter,
>  			remote_addr, rkey, dir);

First I thought this breaks iWarp reads, but they are handled earlier,
which might have been usefu in the commit log.