Re: Kernel fast memory registration API proposal [RFC]

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

> >   - ULP calls a 'rdma_post_close_rkey' helper
> >      * For FRWR this posts the INVALIDATE
> 
> Note: Some send operations automatically invalidate an rkey (and the
> lkey for IB?).  This is intended to avoid having to post the
> invalidate WR explicitly.  Namely IB_WR_READ_WITH_INV and
> IB_WR_SEND_WITH_INV.

IB_WR_RDMA_READ_WITH_INV is only meaningful/implemented for iWarp. Can
you confirm what it does?

 wc.opcode = IB_WR_RDMA_READ_WITH_INV
 wc.sg_list[0].lkey = LKEY;
 wc.rdma.rkey = RKEY;

Does it locally invalidate LKEY or remotely invalidate RKEY? I'm
feeling pretty confident to guess it invalidates LKEY..

'rdma_post_close_rkey' wouldn't work with lkeys, hence the name :)

IB_WR_SEND_WITH_INV is sadly never used in the kernel. It does
invalidate an rkey, and would have to interact with a
'rdma_post_close_rkey'.

A generic flow could be something like:

  if (rdma_post_close_rkey_wc(qp,rkey,wc,...) ..)
    // invalidate triggred by the peer, all done.
  else
    // Peer didn't invalidate, func queued it locally, ULP waits for the next completion, then all done.
  else
    // Error

Where the _wc varient checks the wc to confirm that the requested rkey
has been invalidated.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html