Re: [PATCH v5 00/19] Allocate payload arrays dynamically

Jeff Layton
2025-05-13T12:05:27Z

On Fri, 2025-05-09 at 15:03 -0400, cel@kernel.org wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
>
> In order to make RPCSVC_MAXPAYLOAD larger (or variable in size), we
> need to do something clever with the payload arrays embedded in
> struct svc_rqst and elsewhere.
>
> My preference is to keep these arrays allocated all the time because
> allocating them on demand increases the risk of a memory allocation
> failure during a large I/O. This is a quick-and-dirty approach that
> might be replaced once NFSD is converted to use large folios.
>
> The downside of this design choice is that it pins a few pages per
> NFSD thread (and that's the current situation already). But note
> that because RPCSVC_MAXPAGES is 259, each array is just over a page
> in size, making the allocation waste quite a bit of memory beyond
> the end of the array due to power-of-2 allocator round up. This gets
> worse as the MAXPAGES value is doubled or quadrupled.
>
> This series also addresses similar issues in the socket and RDMA
> transports.
>
> Changes since v4:
> * Replace the use of rq_vec instead of allocating it dynamically
>
> Changes since v3:
> * Improved the rdma_rw context count estimate
> * Dropped "NFSD: Remove NFSSVC_MAXBLKSIZE from .pc_xdrressize"
> * Cleaned up the max size macros a bit
> * Completed the implementation of adjustable max_block_size
>
> Changes since v2:
> * Address Jeff's review comments
> * Address Neil's review comments
> * Start removing a few uses of NFSSVC_MAXBLKSIZE
>
> Chuck Lever (19):
> svcrdma: Reduce the number of rdma_rw contexts per-QP
> sunrpc: Add a helper to derive maxpages from sv_max_mesg
> sunrpc: Remove backchannel check in svc_init_buffer()
> sunrpc: Replace the rq_pages array with dynamically-allocated memory
> sunrpc: Replace the rq_bvec array with dynamically-allocated memory
> NFSD: Use rqstp->rq_bvec in nfsd_iter_read()
> NFSD: De-duplicate the svc_fill_write_vector() call sites
> SUNRPC: Export xdr_buf_to_bvec()
> NFSD: Use rqstp->rq_bvec in nfsd_iter_write()
> SUNRPC: Remove svc_fill_write_vector()
> SUNRPC: Remove svc_rqst :: rq_vec
> sunrpc: Adjust size of socket's receive page array dynamically
> svcrdma: Adjust the number of entries in svc_rdma_recv_ctxt::rc_pages
> svcrdma: Adjust the number of entries in svc_rdma_send_ctxt::sc_pages
> sunrpc: Remove the RPCSVC_MAXPAGES macro
> NFSD: Remove NFSD_BUFSIZE
> NFSD: Remove NFSSVC_MAXBLKSIZE_V2 macro
> NFSD: Add a "default" block size
> SUNRPC: Bump the maximum payload size for the server
>
> fs/nfsd/nfs3proc.c | 5 +-
> fs/nfsd/nfs4proc.c | 10 +--
> fs/nfsd/nfs4state.c | 2 +-
> fs/nfsd/nfs4xdr.c | 2 +-
> fs/nfsd/nfsd.h | 24 +++----
> fs/nfsd/nfsproc.c | 13 ++--
> fs/nfsd/nfssvc.c | 2 +-
> fs/nfsd/nfsxdr.c | 4 +-
> fs/nfsd/vfs.c | 67 ++++++++++++++------
> fs/nfsd/vfs.h | 10 +--
> include/linux/sunrpc/svc.h | 46 ++++++++------
> include/linux/sunrpc/svc_rdma.h | 6 +-
> include/linux/sunrpc/svcsock.h | 4 +-
> net/sunrpc/svc.c | 80 +++++++-----------------
> net/sunrpc/svc_xprt.c | 10 +--
> net/sunrpc/svcsock.c | 15 +++--
> net/sunrpc/xdr.c | 1 +
> net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 8 ++-
> net/sunrpc/xprtrdma/svc_rdma_rw.c | 2 +-
> net/sunrpc/xprtrdma/svc_rdma_sendto.c | 16 ++++-
> net/sunrpc/xprtrdma/svc_rdma_transport.c | 14 +++--
> 21 files changed, 170 insertions(+), 171 deletions(-)

This all looks good to me. You can add this to 8-10 and 18: