Re: [PATCH RFC 0/3] Proposal for exposing rdma_rw MR factor

[Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>]

On Wed, 2017-08-16 at 11:50 -0400, Chuck Lever wrote:
> > On Aug 8, 2017, at 2:28 PM, Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
> > wrote:
> > 
> > 
> > > On Aug 2, 2017, at 3:20 AM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> > > wrote:
> > > 
> > > On Tue, Aug 01, 2017 at 05:06:51PM -0400, Chuck Lever wrote:
> > > > Since I converted NFSD to use the new rdma_rw API, I've been
> > > > struggling with how NFSD determines the size of its send CQ,
> > > > and how
> > > > it provisions its send queue limit (ie, when it waits for more
> > > > send
> > > > queue entries to become available).
> > > > 
> > > > The problem arises because rdma_rw hides the exact number of
> > > > send
> > > > WQEs it has provisioned. It determines this number based on the
> > > > MR
> > > > registration mode (FRWR for iWARP devices, local DMA rkey for
> > > > others), but the mode itself is no longer exposed to ULPs.
> > > > 
> > > > Thus when FRWR is in use, rdma_rw adds more WQEs, but NFSD is
> > > > no
> > > > longer aware of this, does not provision a larger CQ, and does
> > > > not
> > > > raise its send queue accounting limit. That means the extra
> > > > WQEs are
> > > > never really used.
> > > > 
> > > > At LSF this year, Christoph and Sagi proposed a simple API that
> > > > expose information about how many CQ entries are needed so that
> > > > ULPs
> > > > can provision their CQs more accurately.
> > > 
> > > Indeed clean API. Does it make sense to get rid of
> > > sc_max_requests and
> > > use this exported value directly?
> > 
> > I'm going to consider this clean up for a later patch.
> > 
> > So, how shall I submit this new API for upstream? I propose that
> > I submit all three of these via Bruce's tree, with an Acked-by
> > from Doug on the "Add rdma_rw_mr_payload()" patch, if Doug
> > approves.
> 
> Doug?

Your proposal is acceptable.  Ack given for 2nd patch to rdma core.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html