Re: RFC: RPC/RDMA memory invalidation

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Wed, Oct 28, 2015 at 03:56:08PM -0400, Chuck Lever wrote:

> A key question is whether connection loss guarantees that the
> server is fenced, for all device types, from existing
> registered MRs. After reconnect, each MR must be registered
> again before it can be accessed remotely. Is this true for the
> Linux IB core, and all kernel providers, when using FRWR?

MR validation is not linked to a QP in any way. The memory is not
fully fenced until the invalidate completes, or the MR unregister
completes. Nothing else is good enough.

> After a connection loss, the Linux kernel RPC/RDMA client
> creates a new QP as it reconnects, thus I’d expect the QPN to
> be different on the new connection. That should be enough to
> prevent access to MRs that were registered with the previous
> QP and PD, right?

No, the NFS implementation creates a single PD for everything and any
QP in the PD can access all the MRs. This is another security issue of
a different sort.

If there was one PD per QP then the above would be true, since the MR
is linked to the PD.

Even so, moving a QP out of RTR is not a synchronous operation, and
until the CQ is drained, the disoposition of ongoing RDMA is not
defined.

Basically: You can't avoid actually doing a blocking invalidate
operation. The core layer must allow for this if it is going to async
cancel RPCs.

FWIW, the same is true on the send side too, if the RPC had send
buffers and gets canceled, you have to block until a CQ linked to that
send is seen.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html