Re: [PATCH] RDMA/rw: Fix MR pool exhaustion in bvec RDMA READ path

[Chuck Lever <cel@kernel.org>]

On 3/10/26 9:42 AM, Christoph Hellwig wrote:
> On Mon, Mar 09, 2026 at 11:46:21PM -0400, Chuck Lever wrote:
>> Under NFS WRITE workloads the server performs RDMA READs to
>> pull data from the client. With the inflated MR demand, the
>> pool is rapidly exhausted, ib_mr_pool_get() returns NULL, and
>> rdma_rw_init_one_mr() returns -EAGAIN. svcrdma treats this as
>> a DMA mapping failure, closes the connection, and the client
>> reconnects -- producing a cycle of 71% RPC retransmissions and
>> ~100 reconnections per test run. RDMA WRITEs (NFS READ
>> direction) are unaffected because DMA_TO_DEVICE never triggers
>> the max_sgl_rd check.
> 
> So this changelog extensively describes the problem, but it doesn't
> actually say how you fix it.

I didn't want to waste everyone's time, but I can add that.


>> +	 *
>> +	 * TODO: A bulk DMA mapping API for bvecs analogous to
>> +	 * dma_map_sgtable() would provide a proper post-DMA-
>> +	 * coalescing segment count here, enabling the map_wrs
>> +	 * path in more cases.
> 
> This isn't really something the DMA layer can easily do without getting
> as inefficient as the sgtable based path.  What the block layer does
> here is to simply keep a higher level count of merged segments.  The
> other option would be to not create multiple bvecs for continguous
> regions, which is what modern file system do in general, and why the
> above block layer nr_phys_segments based optimization isn't actually
> used all that much these days.

Technically, NFSD isn't a file system, it's a protocol adapter.


> Why can't NFS send a single bvec for contiguous ranges?

Have a look at svc_rdma_build_read_segment(). The RDMA READ path builds
bvecs from rqstp->rq_pages[], which is an array of individual struct
page pointers. Each bvec entry covers at most one page.

This is because I/O payloads arrive in an xdr_buf, which represents its
page data as a struct page ** array (xdr->pages), and svc_rqst::rq_pages
is likewise a flat array of single-page pointers. These pages are
allocated individually (typically from the page allocator via
alloc_pages()), so there's no guarantee of physical contiguity. Even if
adjacent pages happen to be contiguous, the code has no way to know that
without inspecting PFNs (which is exactly what the DMA mapping layer
does).

So currently svcrdma can't send a single bvec for contiguous ranges
because the contiguity information doesn't exist at the NFSD or RPC
layer. Contiguity is (re)discovered only at DMA map time.

The alternative is to build an SGL for mapping the bvec so that rw.c can
get the real contiguity of the pages before proceeding. But that seems
icky.

Long term, I expect that NFSD will need to preserve the folios it gets
from file systems and pass those to the RPC transports without
translating them to an array of page pointers.


-- 
Chuck Lever