[PATCH v4 00/14] Allocate payload arrays dynamically

cel
2025-04-28T19:37:05Z

From: Chuck Lever <chuck.lever@oracle.com>

In order to make RPCSVC_MAXPAYLOAD larger (or variable in size), we
need to do something clever with the payload arrays embedded in
struct svc_rqst and elsewhere.

My preference is to keep these arrays allocated all the time because
allocating them on demand increases the risk of a memory allocation
failure during a large I/O. This is a quick-and-dirty approach that
might be replaced once NFSD is converted to use large folios.

The downside of this design choice is that it pins a few pages per
NFSD thread (and that's the current situation already). But note
that because RPCSVC_MAXPAGES is 259, each array is just over a page
in size, making the allocation waste quite a bit of memory beyond
the end of the array due to power-of-2 allocator round up. This gets
worse as the MAXPAGES value is doubled or quadrupled.

This series also addresses similar issues in the socket and RDMA
transports.

v4 is "code complete", unless there are new code change requests.
I'm not convinced that adding XDR pad alignment to svc_reserve()
is good, but I'm willing to consider it further.

It turns out there is already a tuneable for the maximum read and
write size in NFSD:

/proc/fs/nfsd/max_block_size

Since there is an existing user space API for this, my initial
arguments against adding a tuneable are moot. max_block_size should
be adequate for this purpose, and enabling it to be set to larger
values should not impact the kernel-user space API in any way.

Changes since v3:
* Improved the rdma_rw context count estimate
* Dropped "NFSD: Remove NFSSVC_MAXBLKSIZE from .pc_xdrressize"
* Cleaned up the max size macros a bit
* Completed the implementation of adjustable max_block_size

Changes since v2:
* Address Jeff's review comments
* Address Neil's review comments
* Start removing a few uses of NFSSVC_MAXBLKSIZE

Chuck Lever (14):
svcrdma: Reduce the number of rdma_rw contexts per-QP
sunrpc: Add a helper to derive maxpages from sv_max_mesg
sunrpc: Remove backchannel check in svc_init_buffer()
sunrpc: Replace the rq_pages array with dynamically-allocated memory
sunrpc: Replace the rq_vec array with dynamically-allocated memory
sunrpc: Replace the rq_bvec array with dynamically-allocated memory
sunrpc: Adjust size of socket's receive page array dynamically
svcrdma: Adjust the number of entries in svc_rdma_recv_ctxt::rc_pages
svcrdma: Adjust the number of entries in svc_rdma_send_ctxt::sc_pages
sunrpc: Remove the RPCSVC_MAXPAGES macro
NFSD: Remove NFSD_BUFSIZE
NFSD: Remove NFSSVC_MAXBLKSIZE_V2 macro
NFSD: Add a "default" block size
SUNRPC: Bump the maximum payload size for the server

fs/nfsd/nfs4proc.c | 2 +-
fs/nfsd/nfs4state.c | 2 +-
fs/nfsd/nfs4xdr.c | 2 +-
fs/nfsd/nfsd.h | 24 ++++-------
fs/nfsd/nfsproc.c | 4 +-
fs/nfsd/nfssvc.c | 2 +-
fs/nfsd/nfsxdr.c | 4 +-
fs/nfsd/vfs.c | 2 +-
include/linux/sunrpc/svc.h | 45 +++++++++++++--------
include/linux/sunrpc/svc_rdma.h | 6 ++-
include/linux/sunrpc/svcsock.h | 4 +-
net/sunrpc/svc.c | 51 +++++++++++++++---------
net/sunrpc/svc_xprt.c | 10 +----
net/sunrpc/svcsock.c | 15 ++++---
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 8 +++-
net/sunrpc/xprtrdma/svc_rdma_rw.c | 2 +-
net/sunrpc/xprtrdma/svc_rdma_sendto.c | 16 ++++++--
net/sunrpc/xprtrdma/svc_rdma_transport.c | 14 ++++---
18 files changed, 122 insertions(+), 91 deletions(-)

--
2.49.0