Re: RFC: RPC/RDMA memory invalidation

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Wed, Oct 28, 2015 at 05:30:17PM -0400, Chuck Lever wrote:

> IBTA spec states:
> 
> > MW access operations (i.e. RDMA Write, RDMA Reads, and Atom-
> > ics) are only allowed if the Type 2B MW is in the Valid state and the
> > QP Number (QPN) and PD of the QP performing the MW access op-
> > eration matches the QPN and PD associated with the Bound Type 2B
> > MW.
> 
> Once the QP is out of RTS, there can be no incoming RDMA
> requests that match the R_key, QPN, PD tuple. I think you
> are saying that the QP state change has the same problem
> as not waiting for an invalidation to complete.

MW (Memory Window) is something different from a MR.

MR's do not match on the QPN.

> > If there was one PD per QP then the above would be true, since the MR
> > is linked to the PD.
> 
> There is a per-connection struct rpcrdma_ia that contains
> both a PD and a QP. Therefore there is one PD and only one
> QP (on the client) per connection.

Oh, that is great then

> > FWIW, the same is true on the send side too, if the RPC had send
> > buffers and gets canceled, you have to block until a CQ linked to that
> > send is seen.
> 
> By “you have to block” you mean the send buffer cannot be reused
> until the Send WR is known to have completed, and new Send WRs
> cannot be posted until it is known that enough send queue resources
> are available.

Yes

> I’m not certain we are careful to ensure
> the hardware has truly relinquished the send buffer before it is
> made available for re-use. A known issue.

This is the issue I was thinking of, yes. Ideally the CPU would not
touch the send buffer until the HW is done with it under any
situation. This is less serious than having a rouge writable R_Key
however.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html