Re: [RFC PATCH 1/5] IB/core: Add Core Capability flags to ib_device

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Fri, May 08, 2015 at 08:56:16PM +0000, Hefty, Sean wrote:

> > If you had a device that supported iWARP and RoCE on the same physical
> > link layer, then yes, the app would need a means of saying which
> > transport to use in addition to the type of QP to establish.
> 
> Ah - got it now.  And I agree, there should be some way to specify
> this at the QP level.

Yes, the only way out is to specify on a per QP basis the addressing
and protocol/transport/whatever thing. Socket uses the AF,SOCK,PROTO
tuple to specify this information. We can probably productively use a
similar breakdown:

AF_IB,SOCK_RC,PROTO_IBA  // InfiniBand
AF_OPA,SOCK_RC,PROTO_IBA // Future 32 bit LID OPB 'InfiniBand'
AF_ETH,SOCK_RC,PROTO_IBA // RoCEv1
AF_INET,SOCK_RC,PROTO_IBA // InfiniBand or RoCEv2, depending on the Link Layer
AF_ETH,SOCK_RC,PROTO_USNIC
AF_INET,SOCK_RC,PROTO_USNIC
AF_INET,SOCK_RC,PROTO_IWARP

The expectation would be that any SOCK,PROTO combination can
'interoperate' with any AF combination - which is true with the above
examples, ie with the right magic NAT hardware RoCEv2/v1/IB/(OPA?) can
all interwork at the BTH layer.

Which is to also say, it also unambiguously implies which standard
defines the subtle verbs varietions that the app has to cope with.

Incompatible AH's can not be used with the wrong type of QP, the CM
layers would auto create QPs with the correct type to reflect how the
CM process was run, etc.

But Michael's patches are still an improvement, ie a RoCE + iWarp port
would still have the cap_mad as true and would still have to create a
QP1 interface, which I think is captured OK. The CM side probably
explodes if a port is both iWarp and RoCE, but that is alot of work to
fix anyhow..

I think we have to tackle the addressing problem as part of the RoCEv2
stuff, just burying all this in a qp index is really horrible.

Realistically, today, if a RoCE/iWarp driver appeared then it would
have to present to the system as two RDMA devices.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html