RE: [PATCH V3 1/5] RDMA/core: Transport-independent access flags

Steve Wise
2015-07-14T19:58:23Z

> -----Original Message-----
> From: 'Christoph Hellwig' [mailto:hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org]
> Sent: Tuesday, July 14, 2015 2:45 PM
> To: Steve Wise
> Cc: 'Sagi Grimberg'; 'Steve Wise'; 'Jason Gunthorpe'; 'Tom Talpey'; 'Doug Ledford'; 'Christoph Hellwig'; sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org;
> ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; roid-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; eli-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org; target-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-
> nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org; bfields-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org; 'Oren Duer'
> Subject: Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags
>
> On Tue, Jul 14, 2015 at 02:25:50PM -0500, Steve Wise wrote:
> > if (device_supports_fastreg && device_supports_signature)
> > use FRMR
> > else
> > use DMAMR
> >
> > Shouldn't we just recode it this way?
> >
> > if (device_supports_fastreg)
> > use FRMR
> > else
> > use DMAMR
>
> How does IB_DEVICE_LOCAL_DMA_LKEY (and ->local_dma_lkey) play into
> this? Seems like that should be the preferred option if supported.
>

The local_dma_lkey can be used in any rdma sge that requires an lkey. It is supported for kernel-mode only. So if you're only ever
going to use the lkey for IO, you really don't need a DMA_MR at all, unless you want to somehow partition up your work load by
protection domain. But I claim for lkeys, the PD doesn't really protect anything since the remote peers can't use it anyway.

> Interestingly enough various iWarp driver seem to support this option,
> what's the story behind that? The (to me surprising) conclusion on
> the list was that iWarp would always need a memory regireations that
> also allows remove writes even for lkeys, but from looking at the
> users of IB_DEVICE_LOCAL_DMA_LKEY / local_dma_lkey that seem to
> prefer that over creating a MR.

There is confusion about lkeys and rkeys with regard to iWARP. In the iWARP verbs, there is no distinction between an lkey and
rkey: they are the same key, called a Steering Tag or STAG. When you create a MR, the lkey == rkey == STAG for iwarp transports.
Somewhat related, but really a different issue, is that SGEs that are the target of a read need REMOTE_WRITE access flags on their
STAG for iWARP.

Clear as mud? :)

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in