RE: [PATCH V3 1/5] RDMA/core: Transport-independent access flags

Steve Wise
2015-07-14T20:51:29Z

> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Jason Gunthorpe
> Sent: Tuesday, July 14, 2015 3:42 PM
> To: Steve Wise
> Cc: 'Christoph Hellwig'; 'Sagi Grimberg'; 'Steve Wise'; 'Tom Talpey'; 'Doug Ledford'; sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org;
> roid-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; eli-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org;
> trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org; bfields-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org; 'Oren Duer'
> Subject: Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags
>
> On Tue, Jul 14, 2015 at 02:58:23PM -0500, Steve Wise wrote:
> > The local_dma_lkey can be used in any rdma sge that requires an
> > lkey.
>
> No, this is where iWarp doesn't follow the generic API - a local dma
> lkey cannot be used with iWarp's RDMA_READ WR lkey. In effect RDMA
> READ requires an *rkey* (confusingly stuck into the lkey slot) for
> iWarp. (Right?)
>

Right, a local_dma_lkey is not an rkey, and iwarp requires the rkey for the read destination MR. Further that rkey needs
REMOTE_WRITE.

> *THAT* is really the core difference between IB and iWarp in this
> area, not that the access flags are different.
>

It both. Because an rkey without REMOTE_WRITE would not work.

> (cap_rmda_read_lkey_is_rkey ?)
>

IMO its more like rdma_cap_read_dest_needs_remote_write(). And maybe make it camel step too. ;)

> > domain. But I claim for lkeys, the PD doesn't really protect
> > anything since the remote peers can't use it anyway.
>
> I agree.
>
> To use a PD properly I'd have thought it should be created on a client
> by client basis. The risk is tiny, but client X should not be able to
> guess Y's RKey and then corrupt a data transfer. *Especially* on a
> server if client X hasn't auth'd yet .... That is what the PD is for.
>
> > There is confusion about lkeys and rkeys with regard to iWARP. In
> > the iWARP verbs, there is no distinction between an lkey and rkey:
> > they are the same key, called a Steering Tag or STAG. When you
> > create a MR, the lkey == rkey == STAG for iwarp transports.
> > Somewhat related, but really a different issue, is that SGEs that
> > are the target of a read need REMOTE_WRITE access flags on their
> > STAG for iWARP.
>
> That is the clearest explanation for the iWarp difference I've seen so
> far, thanks!
>
> Christoph: The take away from all this is that on iWarp RDMA_READ
> requires a restricted temporary MR to provide the lkey value in the
> WC. It cannot use local_dma_lkey.
>
> Everything else is the same between IB and iWarp: local_dma_lkey can
> be used as the lkey for SEND, RECV, WRITE.
>

The source of a WRITE can use local_dma_lkey, not the destination. But this is true for IB and IW...

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in