Re: [PATCH v3 05/15] xprtrdma: Remove last ib_reg_phys_mr() call site

[Jason Gunthorpe <jgunthorpe@obsidianresearch.com>]

On Mon, Jul 20, 2015 at 06:31:11PM -0400, Chuck Lever wrote:
> 
> On Jul 20, 2015, at 6:26 PM, Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote:
> 
> > On Mon, Jul 20, 2015 at 03:03:11PM -0400, Chuck Lever wrote:
> >> +	iov->length = size;
> >> +	iov->lkey = ia->ri_have_dma_lkey ?
> >> +				ia->ri_dma_lkey : ia->ri_bind_mem->lkey;
> >> +	rb->rg_size = size;
> >> +	rb->rg_owner = NULL;
> >> 	return rb;
> > 
> > There is something odd looking about this..
> > 
> > ri_bind_mem is only setup in the RPCRDMA_ALLPHYSICAL and
> > RPCRDMA_MTHCAFMR cases.
> > 
> > RPCRDMA_FRMR doesn't set it up.
> > 
> > So this code in rpcrdma_alloc_regbuf is never called for the FRMR
> > case?
> > 
> > If yes, then, how is FRMR working? There is absolutely no reason to
> > use FRMR to register local send buffers, just use the global all
> > memory lkey...
> > 
> > If no, then that is an oops?
> 
> I’ve tested this code, no oops.
> 
> FRWR always uses the DMA lkey. xprtrdma does not use FRWR if
> IB_DEVICE_LOCAL_DMA_LKEY is not asserted.

Ah, I see. Ok.

Is there a reason to link FRWR and LOCAL_DMA_LKEY together? Just use
the code you have for fmr with frwr to get the lkey, probably move it
to rpcrdma_ia_open .. Physical MR should create a 2nd MR dedicated for
rkey use.

That will work really well with the series I'm working on:

https://github.com/jgunthorpe/linux/tree/remove-ib_get_dma_mr

To just drop ib_get_dma_mr entirely.

Jason