[PATCH v2 0/6] Optimize NFSD buffer page management

[Chuck Lever <cel@kernel.org>]

From: Chuck Lever <chuck.lever@oracle.com>

This series solves two problems. First:

 NFSv3 operations have complementary Request and Response sizes.
 When a Request message is large, the corresponding Response
 message is small, and vice versa. The sum of the two message
 sizes is never more than the maximum transport payload size. So
 NFSD could get away with maintaining a single array of pages,
 split between the RPC send and Receive buffer.

 NFSv4 is not as cut and dried. An NFSv4 client may construct an
 NFSv4 COMPOUND that is arbitrarily complex, mixing operations
 that can have large Request size with operations that have a
 large Response size. The resulting server-side buffer size
 requirement can be larger than the maximum transport payload size.

 Therefore we must increase the allocated RPC Call landing zone and
 the RPC Reply construction zone to ensure that arbitrary NFSv4
 COMPOUNDs can be handled.

Second:

 Due to the above, and because NFSD can now handle payload sizes
 considerably larger than 1MB, the number of array entries that
 alloc_bulk_pages() walks through to reset the rqst page arrays
 after each RPC completes has increased dramatically.

 But we observe that the mean size of NFS requests remains smaller
 than a few pages. If only a few pages are consumed while processing
 each RPC, then traversing all of the pages in the page arrays for
 refills is wasted effort. The CPU cost of walking these arrays is
 noticeable in "perf" captures.

 It would be more efficient to keep track of which entries need to
 be refilled, since that is likely to be a small number in the most
 common case, and use alloc_bulk_pages() to fill only those entries.
 
---

Changes since RFC:
- Clarify a number of comments based on review (NeilBrown)
- Possible NFSv3 waste is still open for discussion

Chuck Lever (6):
  SUNRPC: Tighten bounds checking in svc_rqst_replace_page
  SUNRPC: Allocate a separate Reply page array
  SUNRPC: Handle NULL entries in svc_rqst_release_pages
  svcrdma: preserve rq_next_page in svc_rdma_save_io_pages
  SUNRPC: Track consumed rq_pages entries
  SUNRPC: Optimize rq_respages allocation in svc_alloc_arg

 include/linux/sunrpc/svc.h              | 61 +++++++++++++++----------
 net/sunrpc/svc.c                        | 59 +++++++++++++++++-------
 net/sunrpc/svc_xprt.c                   | 47 +++++++++++++++----
 net/sunrpc/svcsock.c                    |  7 +--
 net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 15 ++----
 net/sunrpc/xprtrdma/svc_rdma_rw.c       |  1 +
 net/sunrpc/xprtrdma/svc_rdma_sendto.c   |  6 +--
 7 files changed, 125 insertions(+), 71 deletions(-)

-- 
2.53.0