RE: [PATCH] svcrdma: Cap Read sink allocations at PAGE_ALLOC_COSTLY_ORDER

[Jonathan Flynn <jonathan.flynn@hammerspace.com>]

> -----Original Message-----
> From: Chuck Lever <cel@kernel.org>
> Sent: Saturday, June 6, 2026 9:18 PM
> To: Jonathan Flynn <jonathan.flynn@hammerspace.com>; Mike Snitzer
> <snitzer@kernel.org>
> Cc: linux-nfs@vger.kernel.org; linux-rdma@vger.kernel.org; Chuck Lever
> <chuck.lever@oracle.com>
> Subject: Re: [PATCH] svcrdma: Cap Read sink allocations at
> PAGE_ALLOC_COSTLY_ORDER
>
>
>
> On Sat, Jun 6, 2026, at 1:35 PM, Jonathan Flynn wrote:
> > I tested the PAGE_ALLOC_COSTLY_ORDER change on the same setup.
> > Unfortunately, it did not improve the regression. Throughput was
> > slightly worse than the previous GFP_NOWAIT test, measuring 25.4
GiB/s.
> >
> > Current results are:
> > Original regressed build: ~30.3 GiB/s
> > GFP_NOWAIT build: ~31.0 GiB/s
> > PAGE_ALLOC_COSTLY_ORDER: 25.4 GiB/s
> > Commit reverted: ~73.9 GiB/s
> >
> > I added the results to the shared bundle. (including flamegraph)
> >
> > The GFP_NOWAIT and the Original Commit flamegraphs are nearly
> identical.
> > The dominant stack being:
> > svc_recv()
> > -> svc_rdma_build_read_segment_contig()
> > -> alloc_pages_noprof()
> > -> get_page_from_freelist()
> > -> rmqueue_buddy()
> >
> > The PAGE_ALLOC_COSTLY_ORDER flamegraph is different. Time spent under
> > alloc_pages_noprof() is reduced, but the reduction does not translate
> > into improved throughput.
> >
> > The following percentages were observed:
> >                                                    Original
GFP_NOWAIT
> > COSTLY_ORDER
> > svc_recv()                                 76.09%      75.99%
> > 78.44%
> > alloc_pages_noprof()             58.07%      57.99%
40.29%
> > folios_put_refs()                        7.15%        7.19%
> > 16.06%
> > svc_rdma_read_complete()    7.18%        7.21%               16.08%
> >
> > In other words, the PAGE_ALLOC_COSTLY_ORDER change reduces time
> spent
> > in the allocation path, but a larger fraction of CPU time then appears
> > under
> > svc_rdma_read_complete() and folios_put_refs(), while overall
> > throughput decreases further.
>
> The two failed fixes demonstrate that the current folio allocator is not
up to
> the task -- the problem appears to be on the release side, where the
> individual pages have to be merged back into an order-4 compound page. I
> don't yet see a straightforward way to make it work.
>
> Since we're right up against v7.1-rc7, I've added a patch to nfsd-next
to revert
> 18755b8c2f24 -- it will get pulled back into 7.1.y as soon as the v7.2
merge
> window closes in three weeks.
>
>
> --
> Chuck Lever


This sounds good.

Thank you for taking the time to investigate it and for working through
the test results with us.

If you continue exploring this area in the future and still see promise in
the contiguous allocation approach, I'd be happy to help test additional
changes as time permits.

-Jon