[PATCHv7 0/9] RoCEE support to Infiniband

[PATCHv7 0/9] RoCEE support to Infiniband

[Eli Cohen]

RoCEE (pronounced ‘rocky’) allows running the IB transport protocol
using Ethernet frames, enabling the deployment of IB semantics on
lossless Ethernet fabrics.

RoCEE packets are standard Ethernet frames with an IEEE assigned
Ethertype, a GRH, unmodified IB transport headers and payload.  IB
subnet management and SA services are not required for RoCEE
operation; Ethernet management practices are used instead. RoCEE
encodes IP addresses into its GIDs and resolves MAC addresses using
the host IP stack. For multicast GIDs, standard IP to MAC mappings
apply.

The OFA RDMA Verbs API is syntactically unmodified. The CMA is adapted
to support RoCEE ports allowing existing RDMA applications to run over
RoCEE with no changes.

Address handles for RoCEE are required to contain valid L3 addresses
(GIDs) and the IB L2 address fields become reserved. The complementary
Ethernet L2 address information is subsequently resolved below the
API. 

As there is no SA in RoCEE, the CMA code is adapted to locally fill-in
corresponding path record attributes for RoCEE address handles. Also,
the CMA provides the required address handle attributes for SIDR
requests and joining of multicast groups. 

With this patch set, a RoCEE port is currently assigned a single GID,
encoding the IPv6 link-local address of the corresponding netdev; the
CMA RoCEE code temporarily uses these IPv6 link-local addresses as
GIDs instead of the IP address provided by the user. 

Support for host VLAN tagged packets and setting of the user prio on
RoCEE frames will be added in the near future. Also, with these
patches, RoCEE multicast frames may be broadcast as there is currently
no use of a L2 multicast group membership protocol.

To enable RoCEE with the mlx4 driver stack, both the mlx4_en and
mlx4_ib drivers must be loaded, and the netdevice for the
corresponding RoCEE port must be running. Individual ports of a multi
port HCA can be independently configured as Ethernet (with support for
RoCEE) or IB, as is already the case.

We have successfully tested MPI, SDP, RDS, and native Verbs
applications over RoCEE.

Following is a series of 9 patches based on Roland's for-next branch.
This new series reflects changes based on feedback from the community
on the previous patch set.

Changes from v6:
1. Rebase on 2.6.32-rc6.
2. Fix loopback support.
3. Undo ABI version bumping.
4. Use the notion of 'link layer' attribute for ports instead of 'port
transport'.


Signed-off-by: Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
---

 drivers/infiniband/core/agent.c           |   37 +-
 drivers/infiniband/core/cma.c             |  261 +++++++++++++++
 drivers/infiniband/core/mad.c             |   66 ++-
 drivers/infiniband/core/multicast.c       |   25 +
 drivers/infiniband/core/sa_query.c        |   39 +-
 drivers/infiniband/core/ucma.c            |   45 ++
 drivers/infiniband/core/ud_header.c       |  111 ++++++
 drivers/infiniband/core/user_mad.c        |   11 
 drivers/infiniband/core/uverbs.h          |    1 
 drivers/infiniband/core/uverbs_cmd.c      |   32 +
 drivers/infiniband/core/uverbs_main.c     |    1 
 drivers/infiniband/core/verbs.c           |   25 +
 drivers/infiniband/hw/mlx4/ah.c           |  178 ++++++++--
 drivers/infiniband/hw/mlx4/mad.c          |   32 +
 drivers/infiniband/hw/mlx4/main.c         |  497 +++++++++++++++++++++++++++---
 drivers/infiniband/hw/mlx4/mlx4_ib.h      |   34 +-
 drivers/infiniband/hw/mlx4/qp.c           |  194 ++++++++---
 drivers/infiniband/ulp/ipoib/ipoib_main.c |    5 
 drivers/net/mlx4/en_main.c                |   15 
 drivers/net/mlx4/en_port.c                |    4 
 drivers/net/mlx4/en_port.h                |    3 
 drivers/net/mlx4/fw.c                     |    3 
 drivers/net/mlx4/intf.c                   |   20 +
 drivers/net/mlx4/main.c                   |    6 
 drivers/net/mlx4/mlx4.h                   |    1 
 include/linux/mlx4/cmd.h                  |    1 
 include/linux/mlx4/device.h               |   31 +
 include/linux/mlx4/driver.h               |   16 
 include/linux/mlx4/qp.h                   |    6 
 include/rdma/ib_addr.h                    |   98 +++++
 include/rdma/ib_pack.h                    |   26 +
 include/rdma/ib_user_verbs.h              |   19 +
 include/rdma/ib_verbs.h                   |   24 +
 33 files changed, 1639 insertions(+), 228 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCHv7 0/9] RoCEE support to Infiniband

[Roland Dreier]

 > RoCEE (pronounced ‘rocky’) allows running the IB transport protocol
 > using Ethernet frames, enabling the deployment of IB semantics on
 > lossless Ethernet fabrics.

Just starting to review this, but let me say at the start that I really
don't like the term "RoCEE" for this stuff.  The OFED 1.5 release you
guys did was called "RDMAoE" but I really think "IBoE" is the best term.

Taking the two halves of "RoCEE": The "CEE" acronym doesn't seem
appropriate, since the standard is really called "DCB" and in any case
these patches (as far as I can tell) don't use any DCB features.  "E"
for Ethernet would make more sense.  And "R" or "RDMA" seems
insufficiently descriptive -- the whole point of this protocol is that
it uses the IB transport layer rather than iWARP or something else, so
it seems much clearer to call this "IB".  Plain "R" is really opaque --
who would ever guess that "R" stands for "RDMA".

Furthermore, it makes sense to me to take advantage of all the marketing
and exposition that we did for FCoE -- which was not called "SCSIoDCB".
The parallels are nearly exact, and being able to say that "IBoE does
is to IB transport as FCoE is to Fibre Channel" is a very clear analogy.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCHv7 0/9] RoCEE support to Infiniband

[Liran Liss]

RoCEE was chosen for these patches in accordance with the terminology
used in the proposal discussed by the OFA.  We have no problem
resubmitting these patches using "iboe" so that you can push them into
2.6.34.  We will need to stay aware of any future name recommendations
by relevant industry bodies.
--Liran
 

-----Original Message-----
From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
[mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Roland Dreier
Sent: Tuesday, January 12, 2010 7:12 PM
To: Eli Cohen
Cc: Linux RDMA list
Subject: Re: [PATCHv7 0/9] RoCEE support to Infiniband


 > RoCEE (pronounced 'rocky') allows running the IB transport protocol
> using Ethernet frames, enabling the deployment of IB semantics on  >
lossless Ethernet fabrics.

Just starting to review this, but let me say at the start that I really
don't like the term "RoCEE" for this stuff.  The OFED 1.5 release you
guys did was called "RDMAoE" but I really think "IBoE" is the best term.

Taking the two halves of "RoCEE": The "CEE" acronym doesn't seem
appropriate, since the standard is really called "DCB" and in any case
these patches (as far as I can tell) don't use any DCB features.  "E"
for Ethernet would make more sense.  And "R" or "RDMA" seems
insufficiently descriptive -- the whole point of this protocol is that
it uses the IB transport layer rather than iWARP or something else, so
it seems much clearer to call this "IB".  Plain "R" is really opaque --
who would ever guess that "R" stands for "RDMA".

Furthermore, it makes sense to me to take advantage of all the marketing
and exposition that we did for FCoE -- which was not called "SCSIoDCB".
The parallels are nearly exact, and being able to say that "IBoE does is
to IB transport as FCoE is to Fibre Channel" is a very clear analogy.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info
at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCHv7 0/9] RoCEE support to Infiniband

[Roland Dreier]

 > RoCEE was chosen for these patches in accordance with the terminology
 > used in the proposal discussed by the OFA.  We have no problem
 > resubmitting these patches using "iboe" so that you can push them into
 > 2.6.34.  We will need to stay aware of any future name recommendations
 > by relevant industry bodies.

Well, don't just take my opinion as the final answer.  It would be good
if someone could "officially" pick a name.  However I really do feel
that RoCEE is a pretty bad choice, since it is both technically
inaccurate and also hard for users to understand.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html