Re: [RFC PATCH 1/5] IB/core: Add Core Capability flags to ib_device

[Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>]

[-- Attachment #1: Type: text/plain, Size: 3548 bytes --]

On Tue, 2015-05-05 at 19:27 +0000, Liran Liss wrote:
> > From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> 
> > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Hefty, Sean
> > >
> > > In accordance with what we've been talking about, drop IBOE for ROCE.
> > >
> > > Drop the UDP off of USNIC, then define a bit for CAP_PROT_UDP_ENCAP.
> > > USNIC will be just USNIC, USNIC_UDP will be USNIC | UDP_ENCAP, ROCE v1
> > > will be ROCE, and ROCEv2 will be ROCE | UDP_ENCAP.
> > 
> > USNIC_UDP is just UDP.  I don't understand why we would want 'USNIC |
> > UDP_ENCAP', or what UDP_ENCAP is intended to convey.  Nothing is being
> > encapsulated.
> > 
> 
> I agree that the UDP_ENCAP notion is confusing.
> We should stick to ROCE_V1 and ROCE_V2.
> 
> > RoCEv2 is IB transport over UDP.
> > 
> > I'm not sure what the protocol field is intended to imply.  If we want to
> > expose the link, network, transport, and RDMA protocols in use, shouldn't
> > these be separate fields or bits?  And even then, I'm not sure what use this
> > has for the ULPs.  iWarp does not require Ethernet or TCP.  RoCEv2 would
> > work fine over any link.  And the core layer should not assume that a device is
> > limited to supporting only one protocol, especially at the network and
> > transport levels.  I vote for deprecating the protocol goofiness.
> > 
> > - Sean
> 
> The protocol notion might not have any value for ULPs, but it is useful for core
> management code. Also, when we extend these notions to user-space, admins will
> actually want to know what wire protocols a certain device can support, even
> just for the sake of interoperability.

So I've been thinking a bit more about the overall architecture of the
underlying patch set from Michael.  The structure of that patch set is
to a certain extent dictating this follow on patch set and so any
problems in it end up being transmitted here as well.

Sean's comments about transports I think were spot on, and they got me
doing a little thought exercise in my head.

Theoretically, if we were to take the SoftiWARP and SoftRoCE drivers, we
could merge them down to one driver that supported both iWARP and RoCE
(and we could add USNIC too I suspect).  Because all of these things
work on an Ethernet link layer, there is no reason they couldn't all be
presented on the same device.  So if you had a single verbs device that
could support all of these things, where would you need to capture and
store the relevant information?

This is what I have so far:

On a per device basis: not much to be honest, node_type IB_CA/RNIC for
back compatibility and that's pretty much it

On a per port basis: link layer (Ethernet, IB, or OPA).  Notice that I
didn't include the transport here.  Two of the above link layers
strictly imply their transport.  The third link layer could have
multiple transports.  Which brings me to my next level...

On a per queue pair basis: if (link_layer == Ethernet) iWARP/RoCE/USNIC
                         if (qp_type == RoCE && qp != UD) RoCEv1/RoCEv2

On a per ah basis: if (qp_type == RoCE && qp = UD) RoCEv1/RoCEv2

Michael's patch set was intended to give us a framework within which we
can start cleaning things up.  Once we get to cleaning things up, I
think the above is where we should target storing the relevant
information.

Thoughts?


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]