Re: [PATCH v5 00/21] NFS/RDMA client patches for 3.17

[Anna Schumaker <Anna.Schumaker@netapp.com>]

I've applied v5 and updated my tree.  Thanks, Chuck!

Anna

On 07/29/2014 05:23 PM, Chuck Lever wrote:
> The main purpose of this series is to address connection drop
> recovery issues by fixing FRMR re-use to make it less likely the
> client will deadlock due to a memory management operation error.
> Some clean-ups and other fixes are present as well.
> 
> See topic branch nfs-rdma-for-3.17-plan-b in
> 
> git://git.linux-nfs.org/projects/cel/cel-2.6.git
> 
> Topic branch nfs-rdma-for-3.17 still contains v4 of this series.
> You can choose whichever series is convenient for you and Roland.
> 
> I tested with NFSv3 and NFSv4 on all three supported memory
> registration modes. Used cthon04, iozone, vdbench, and dbench with
> both Solaris and Linux NFS/RDMA servers. Used xfstests with Linux.
> 
> 
> v5:
> 
> - Rebased on v3.16-rc7
> 
> - Replaced patch 08/21 with an equivalent one-liner that does not
>   alter include/rdma/ib_verbs.h. A couple of subsequent patches were
>   updated to apply on the new 08/21. This allows the whole series to
>   be submitted through the NFS tree.
> 
> 
> v4:
> 
> - Rebased on v3.16-rc6
> 
> - Add Tested-by: from Shirley Ma and Devesh Sharma
> 
> 
> v3:
> Only two substantive changes:
> 
> - Patch 08/21 now uses generic IB helpers for managing FRMR
>   rkeys
> 
> - Add Tested-by: from Steve Wise
> 
> 
> v2:
> Many patches from v1 have been written or replaced.
> 
> The MW ref counting approach in v1 is abandoned. Instead, I've
> eliminated signaling FAST_REG_MR and LOCAL_INV, and added
> appropriate recovery mechanisms after a transport reconnect that
> should prevent rkey dis-synchrony entirely.
> 
> A couple of optimizations have been added, including:
> 
> - Allocating each MW separately rather than carving each out of a
>   large piece of contiguous memory
> 
> - Now that the receive CQ upcall handler dequeues a bundle of CQEs
>   at once, fire off the reply handler tasklet just once per upcall
>   to reduce context switches and how often hard IRQs are disabled
> 
> ---
> 
> Chuck Lever (21):
>       xprtrdma: Fix panic in rpcrdma_register_frmr_external()
>       xprtrdma: Protect ia->ri_id when unmapping/invalidating MRs
>       xprtrdma: Limit data payload size for ALLPHYSICAL
>       xprtrdma: Update rkeys after transport reconnect
>       xprtrdma: On disconnect, don't ignore pending CQEs
>       xprtrdma: Don't invalidate FRMRs if registration fails
>       xprtrdma: Unclutter struct rpcrdma_mr_seg
>       xprtrdma: Back off rkey when FAST_REG_MR fails
>       xprtrdma: Chain together all MWs in same buffer pool
>       xprtrdma: Properly handle exhaustion of the rb_mws list
>       xprtrdma: Reset FRMRs when FAST_REG_MR is flushed by a disconnect
>       xprtrdma: Reset FRMRs after a flushed LOCAL_INV Work Request
>       xprtrdma: Don't post a LOCAL_INV in rpcrdma_register_frmr_external()
>       xprtrdma: Disable completions for FAST_REG_MR Work Requests
>       xprtrdma: Disable completions for LOCAL_INV Work Requests
>       xprtrdma: Rename frmr_wr
>       xprtrdma: Allocate each struct rpcrdma_mw separately
>       xprtrdma: Schedule reply tasklet once per upcall
>       xprtrdma: Make rpcrdma_ep_disconnect() return void
>       xprtrdma: Remove RPCRDMA_PERSISTENT_REGISTRATION macro
>       xprtrdma: Handle additional connection events
> 
> 
>  include/linux/sunrpc/xprtrdma.h |    2 
>  net/sunrpc/xprtrdma/rpc_rdma.c  |   83 ++--
>  net/sunrpc/xprtrdma/transport.c |   17 +
>  net/sunrpc/xprtrdma/verbs.c     |  736 ++++++++++++++++++++++++++-------------
>  net/sunrpc/xprtrdma/xprt_rdma.h |   61 +++
>  5 files changed, 601 insertions(+), 298 deletions(-)
> 
> --
> Chuck Lever
>