Re: [PATCH v2 00/18] NFS/RDMA client patches for v4.7

[Chuck Lever <chuck.lever@oracle.com>]

> On Apr 26, 2016, at 10:13 AM, Steve Wise <swise@opengridcomputing.com> wrote:
> 
> Hey Chuck, I'm testing this series on cxgb4.    I'm running 'iozone -a -+d -I' on a share and watching the server stats.  Are the starve numbers expected?

Yes, unless you're seeing much higher numbers than
you used to.


> Every 5.0s: for s in  /proc/sys/sunrpc/svc_rdma/rdma_* ; do echo -n "$(basename $s): "; cat $s; done                              Tue Apr 26 07:10:17 2016
> 
> rdma_stat_read: 379872
> rdma_stat_recv: 498144
> rdma_stat_rq_poll: 0
> rdma_stat_rq_prod: 0
> rdma_stat_rq_starve: 675564

This means work was enqueued on the svc_xprt, but by the
time the upper layer invoked svc_rdma_recvfrom, the work
was already handled by an earlier wake-up.

I'm not exactly sure why this happens, but it seems to be
normal (if suboptimal).


> rdma_stat_sq_poll: 0
> rdma_stat_sq_prod: 0
> rdma_stat_sq_starve: 1748000

No SQ space to post a Send, so the caller is put to sleep.

The server chronically underestimates the SQ depth, especially
for FRWR. I haven't figured out a better way to estimate it.

But it's generally harmless, as there is a mechanism to put
callers to sleep until there is space on the SQ.


> rdma_stat_write: 2805420
> 
> 
> On 4/25/2016 2:20 PM, Chuck Lever wrote:
>> Second version of NFS/RDMA client patches proposed for merge into
>> v4.7. Thanks in advance for any review comments!
>> 
>> Attempt to fence memory regions after a signal interrupts a
>> synchronous RPC. This prevents a server from writing a reply into a
>> client's memory after the memory has been released due to a signal.
>> 
>> Support providing a Read list and Reply chunk together in one RPC
>> call. This is a pre-requisite for using krb5i or krb5p on RPC/RDMA.
>> 
>> In addition, the following changes and fixes are included:
>> 
>> - Use new ib_drain_qp() API
>> - Advertise max size of NFSv4.1 callbacks on RPC/RDMA
>> - Prevent overflowing the server's receive buffers
>> - Send small NFS WRITEs inline rather than using a Read chunk
>> - Detect connection loss sooner
>> 
>> 
>> Available in the "nfs-rdma-for-4.7" topic branch of this git repo:
>> 
>> git://git.linux-nfs.org/projects/cel/cel-2.6.git
>> 
>> Or for browsing:
>> 
>> http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=log;h=refs/heads/nfs-rdma-for-4.7
>> 
>> 
>> Changes since v1:
>> - Rebased on v4.6-rc5
>> - Updated patch description for "Avoid using Write list for ..."
>> 
>> ---
>> 
>> Chuck Lever (18):
>>       sunrpc: Advertise maximum backchannel payload size
>>       xprtrdma: Bound the inline threshold values
>>       xprtrdma: Limit number of RDMA segments in RPC-over-RDMA headers
>>       xprtrdma: Prevent inline overflow
>>       xprtrdma: Avoid using Write list for small NFS READ requests
>>       xprtrdma: Update comments in rpcrdma_marshal_req()
>>       xprtrdma: Allow Read list and Reply chunk simultaneously
>>       xprtrdma: Remove rpcrdma_create_chunks()
>>       xprtrdma: Use core ib_drain_qp() API
>>       xprtrdma: Rename rpcrdma_frwr::sg and sg_nents
>>       xprtrdma: Save I/O direction in struct rpcrdma_frwr
>>       xprtrdma: Reset MRs in frwr_op_unmap_sync()
>>       xprtrdma: Refactor the FRWR recovery worker
>>       xprtrdma: Move fr_xprt and fr_worker to struct rpcrdma_mw
>>       xprtrdma: Refactor __fmr_dma_unmap()
>>       xprtrdma: Add ro_unmap_safe memreg method
>>       xprtrdma: Remove ro_unmap() from all registration modes
>>       xprtrdma: Faster server reboot recovery
>> 
>> 
>>  fs/nfs/nfs4proc.c                  |   10 -
>>  include/linux/sunrpc/clnt.h        |    1
>>  include/linux/sunrpc/xprt.h        |    1
>>  include/linux/sunrpc/xprtrdma.h    |    4
>>  net/sunrpc/clnt.c                  |   17 +
>>  net/sunrpc/xprtrdma/backchannel.c  |   16 +
>>  net/sunrpc/xprtrdma/fmr_ops.c      |  134 +++++++--
>>  net/sunrpc/xprtrdma/frwr_ops.c     |  214 ++++++++-------
>>  net/sunrpc/xprtrdma/physical_ops.c |   39 ++-
>>  net/sunrpc/xprtrdma/rpc_rdma.c     |  517 ++++++++++++++++++++++--------------
>>  net/sunrpc/xprtrdma/transport.c    |   16 +
>>  net/sunrpc/xprtrdma/verbs.c        |   91 ++----
>>  net/sunrpc/xprtrdma/xprt_rdma.h    |   42 ++-
>>  net/sunrpc/xprtsock.c              |    6
>>  14 files changed, 674 insertions(+), 434 deletions(-)
>> 
>> --
>> Chuck Lever
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Chuck Lever