Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags

Sagi Grimberg
2015-07-14T09:22:14Z

On 7/14/2015 10:37 AM, 'Christoph Hellwig' wrote:
> On Mon, Jul 13, 2015 at 03:36:44PM -0400, Tom Talpey wrote:
>> On 7/11/2015 6:25 AM, 'Christoph Hellwig' wrote:
>>> I think what we need to support for now is FRMR as the primary target,
>>> and FMR as a secondar[y].
>>
>> FMR is a *very* bad choice, for several reasons.
>>
>> 1) It's not supported by very many devices, in fact it might even
>> be thought to be obsolete.
>
> It's support by the Mellanox adapters,

*Older* Mellanox adapters. mlx5 (and future drivers I assume) will
no longer support FMRs.

> which might be a minority of
> the drivers, but at least in the field I've been in the aboslute
> majority of the deployed hardware.
>
> It's the default in the SRP initiator and iSER initiators, and the
> primary fallback in the NFS client if FRs aren't available.

We should gradually move away from that.

>
> I don't claim to be an expert on memory registrations models, as they
> are horribly, horrible documents (baiscally not at all in the source
> tree), so my knowledge is from looking at and using implementations as
> well as this useful writeup from the NFS folks:
>
> http://wiki.linux-nfs.org/wiki/index.php/NfsRdmaClient/MemRegModes
>
> Which misses or downplays an important restriction of PHYS
> registrations: They dont allow coalescing non-contiguous memory
> into a single maping, which makes them totally unsuitable for iSER
> which only allows a single rkey/offset/len pair and requires
> additional RDMA READ/WRITE roundtrips and larger S/G lists for other
> protocols.

I'm willing to add a proper memory_registration.txt under
Documentation/infiniband/ which would capture the items that
were brought up here.

>
> Based on looking at the consumers and the above table I think FMR
> are still the better fallback for devices that don't support FR,

It's better if you want it fast. I can't stress it enough, but IMO, the
fallback should *not* be in the API, but rather in the ULP.
Ideally, at some point it won't need to fall back, and we can remove
the API.

> but supporting PHYS MRs is easy enough that adding it to a common
> layer seems easy enough if some cares enough to regularly test it.
>

Someone needs a good reason to use it.