Re: [PATCH] RDMA/rw: Fix MR pool exhaustion in bvec RDMA READ path

[Chuck Lever <cel@kernel.org>]

On 3/10/26 2:37 PM, Leon Romanovsky wrote:
> On Tue, Mar 10, 2026 at 10:36:43AM -0400, Chuck Lever wrote:
>> On 3/10/26 9:42 AM, Christoph Hellwig wrote:
>>> On Mon, Mar 09, 2026 at 11:46:21PM -0400, Chuck Lever wrote:
>>>> Under NFS WRITE workloads the server performs RDMA READs to
>>>> pull data from the client. With the inflated MR demand, the
>>>> pool is rapidly exhausted, ib_mr_pool_get() returns NULL, and
>>>> rdma_rw_init_one_mr() returns -EAGAIN. svcrdma treats this as
>>>> a DMA mapping failure, closes the connection, and the client
>>>> reconnects -- producing a cycle of 71% RPC retransmissions and
>>>> ~100 reconnections per test run. RDMA WRITEs (NFS READ
>>>> direction) are unaffected because DMA_TO_DEVICE never triggers
>>>> the max_sgl_rd check.
>>>
>>> So this changelog extensively describes the problem, but it doesn't
>>> actually say how you fix it.
>>
>> I didn't want to waste everyone's time, but I can add that.
>>
>>
>>>> +	 *
>>>> +	 * TODO: A bulk DMA mapping API for bvecs analogous to
>>>> +	 * dma_map_sgtable() would provide a proper post-DMA-
>>>> +	 * coalescing segment count here, enabling the map_wrs
>>>> +	 * path in more cases.
>>>
>>> This isn't really something the DMA layer can easily do without getting
>>> as inefficient as the sgtable based path.  What the block layer does
>>> here is to simply keep a higher level count of merged segments.  The
>>> other option would be to not create multiple bvecs for continguous
>>> regions, which is what modern file system do in general, and why the
>>> above block layer nr_phys_segments based optimization isn't actually
>>> used all that much these days.
>>
>> Technically, NFSD isn't a file system, it's a protocol adapter.
>>
>>
>>> Why can't NFS send a single bvec for contiguous ranges?
>>
>> Have a look at svc_rdma_build_read_segment(). The RDMA READ path builds
>> bvecs from rqstp->rq_pages[], which is an array of individual struct
>> page pointers. Each bvec entry covers at most one page.
>>
>> This is because I/O payloads arrive in an xdr_buf, which represents its
>> page data as a struct page ** array (xdr->pages), and svc_rqst::rq_pages
>> is likewise a flat array of single-page pointers. These pages are
>> allocated individually (typically from the page allocator via
>> alloc_pages()), so there's no guarantee of physical contiguity. Even if
>> adjacent pages happen to be contiguous, the code has no way to know that
>> without inspecting PFNs (which is exactly what the DMA mapping layer
>> does).
>>
>> So currently svcrdma can't send a single bvec for contiguous ranges
>> because the contiguity information doesn't exist at the NFSD or RPC
>> layer. Contiguity is (re)discovered only at DMA map time.
>>
>> The alternative is to build an SGL for mapping the bvec so that rw.c can
>> get the real contiguity of the pages before proceeding. But that seems
>> icky.
>>
>> Long term, I expect that NFSD will need to preserve the folios it gets
>> from file systems and pass those to the RPC transports without
>> translating them to an array of page pointers.
> 
> Folio sounds like a correct approach to me, why do you mark it as "long term"?

All four NFS maintainers are aware of this need, and I /think/ we are
all on board with the "folio/bvec" approach. But there are a number of
reasons this is not just a "go write the code" kind of problem:

- xdr_buf is used by both the NFS client and NFS server stacks, and they
are separately maintained.

- xdr_buf is used from nearly the top to the bottom of these stacks, so
making this kind of change will be painstaking.

- The XDR layer is full of helper APIs that deal with xdr_buf page
arrays that would need attention.

- I need to understand whether addressing the DMA map problem has
benefits for more broadly-deployed RPC transports such as TCP.


I'll have to think if we can just add something clever to the xdr_buf
that only NFSD can use and then have that act as the conduit from file
system to RPC transport. That could act as a prototype.


-- 
Chuck Lever