Upstream support for multicast IBoE

Upstream support for multicast IBoE

[Shawn Bohrer]

It appears the upstream libmlx4/libibverbs currently do not support
multicast IBoE traffic.  For example:

$ mckey -m 239.1.1.1 -b 10.0.202.3
mckey: starting server
mckey: joining
mckey: joined dgid: ff0e::ffff:ef01:101 mlid 0x0 sl 0
mckey: failure creating address handle
test complete
return status 0

Debugging this I can see it fails in libmlx4 in mlx4_create_ah().  In
this case mlx4_resolve_grh_to_l2() returns 1.  Comparing with the OFED
libmlx4/libibverbs I've found the relevant differences are in the
following patch:

http://git.openfabrics.org/git?p=ofed_1_5/libmlx4.git;a=blob;f=fixes/rocee_add_support.patch;h=017b075d639aa2f349424ec94e0089de631470fd;hb=HEAD

This OFED patch seems to partially overlap and conflict with upstream
libmlx4 commits:

902aa188c156e8e2832f5afdf76eb2655fa80582 Add IBoE support
3e73678e88d2ea414a9f4ae187f6412593bd0a1f Add IBoE UD/VLANs support

In general it is very unclear to me what the development work flow is
between the OFED and upstream versions of these libraries.  Is there a
plan to add multicast IBoE support to the upstream versions?

Thanks,
Shawn


---------------------------------------------------------------
This email, along with any attachments, is confidential. If you 
believe you received this message in error, please contact the 
sender immediately and delete all copies of the message.  
Thank you.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Upstream support for multicast IBoE

[Roland Dreier]

On Tue, Nov 22, 2011 at 2:38 PM, Shawn Bohrer <sbohrer-EgGFQ3RFNTIP7C3xziwOQw@public.gmane.org> wrote:
> $ mckey -m 239.1.1.1 -b 10.0.202.3
> mckey: starting server
> mckey: joining
> mckey: joined dgid: ff0e::ffff:ef01:101 mlid 0x0 sl 0
> mckey: failure creating address handle
> test complete
> return status 0
>
> Debugging this I can see it fails in libmlx4 in mlx4_create_ah().  In
> this case mlx4_resolve_grh_to_l2() returns 1.

Or, looks like the libmlx4 patches I applied from you are missing
multicast AH support?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Upstream support for multicast IBoE

[Jason Gunthorpe]

On Tue, Nov 22, 2011 at 03:44:01PM -0800, Roland Dreier wrote:
> On Tue, Nov 22, 2011 at 2:38 PM, Shawn Bohrer <sbohrer-EgGFQ3RFNTIP7C3xziwOQw@public.gmane.org> wrote:
> > $ mckey -m 239.1.1.1 -b 10.0.202.3
> > mckey: starting server
> > mckey: joining
> > mckey: joined dgid: ff0e::ffff:ef01:101 mlid 0x0 sl 0
> > mckey: failure creating address handle
> > test complete
> > return status 0
> >
> > Debugging this I can see it fails in libmlx4 in mlx4_create_ah(). ?In
> > this case mlx4_resolve_grh_to_l2() returns 1.
> 
> Or, looks like the libmlx4 patches I applied from you are missing
> multicast AH support?

I thought that was deliberate because of the lack of specification and
the strange reliance on IPv6 MLD for IBoE packet types...

AFAIK none of that has changed?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Upstream support for multicast IBoE

[Roland Dreier]

On Tue, Nov 22, 2011 at 4:07 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> I thought that was deliberate because of the lack of specification and
> the strange reliance on IPv6 MLD for IBoE packet types...
>
> AFAIK none of that has changed?

I honestly can't remember the discussion.  But we have the ad hoc
MGID -> ethernet address mapping in the kernel, did we really decide
just to leave it out of userspace?

The current situation is pretty embarassing -- is the IBTA still pretending
no standard needs to be written?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Upstream support for multicast IBoE

[Or Gerlitz]

Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> looks like the libmlx4 patches I applied from you are missing multicast AH support?

Yes, when I reworked the patches for submission I went from easier
(RC) to harder (ucast UD + Vlans) and eventually didn't nailed down
the multicast front. I probably planned getting there on a later point
but it remained behind, anyway, if you're okay to have the "ad hoc
MGID -> ethernet address mapping" as you commented on this thread, I
can make the patch to add that.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Upstream support for multicast IBoE

[Or Gerlitz]

> Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
>> looks like the libmlx4 patches I applied from you are missing multicast AH support?

On a related note, what/when would it take to get a release of
libibverbs/libmlx4? so far there's no single release that supports
IBoE and people who don't go OFED have nothing to to use (sure, they
can clone the gits...)  IBoE wise.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Upstream support for multicast IBoE

[Shawn Bohrer]

On Wed, Nov 23, 2011 at 11:17:02PM +0200, Or Gerlitz wrote:
> Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> > looks like the libmlx4 patches I applied from you are missing multicast AH support?
> 
> Yes, when I reworked the patches for submission I went from easier
> (RC) to harder (ucast UD + Vlans) and eventually didn't nailed down
> the multicast front. I probably planned getting there on a later point
> but it remained behind, anyway, if you're okay to have the "ad hoc
> MGID -> ethernet address mapping" as you commented on this thread, I
> can make the patch to add that.

Is there any estimate on when we might see something like this
upstream?

Thanks,
Shawn


---------------------------------------------------------------
This email, along with any attachments, is confidential. If you 
believe you received this message in error, please contact the 
sender immediately and delete all copies of the message.  
Thank you.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Upstream support for multicast IBoE

[Or Gerlitz]

Shawn Bohrer <sbohrer-EgGFQ3RFNTIP7C3xziwOQw@public.gmane.org> wrote:
> Is there any estimate on when we might see something like this upstream?

Could you elaborate a little on your use case for multicast IBoE
traffic? e.g how the setup looks like and how are your Ethernet
switches act to route that traffic.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Upstream support for multicast IBoE

[Shawn Bohrer]

On Wed, Jan 11, 2012 at 09:49:25PM +0200, Or Gerlitz wrote:
> Shawn Bohrer <sbohrer-EgGFQ3RFNTIP7C3xziwOQw@public.gmane.org> wrote:
> > Is there any estimate on when we might see something like this upstream?
> 
> Could you elaborate a little on your use case for multicast IBoE
> traffic? e.g how the setup looks like and how are your Ethernet
> switches act to route that traffic.

I'm not sure exactly what you are asking here.  We do what I would
imagine is a typical one to many UD multicast.  We code directly to
libibvers and librdmacm, and everything is sent IBoE.

The hosts are in a spine/leaf configuration and all traffic is sent
over vlans.  My understanding is that the multicast IBoE traffic is
simply sent as broadcast and that the adapters do the necessary
filtering.

Really from my point of view OFED already does what we want, but I
would really like to see this supported upstream.

Thanks,
Shawn


---------------------------------------------------------------
This email, along with any attachments, is confidential. If you 
believe you received this message in error, please contact the 
sender immediately and delete all copies of the message.  
Thank you.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Upstream support for multicast IBoE

[Shawn Bohrer]

Hi Or,

On Thu, Jan 12, 2012 at 11:01:58AM -0600, Shawn Bohrer wrote:
> On Wed, Jan 11, 2012 at 09:49:25PM +0200, Or Gerlitz wrote:
> > Shawn Bohrer <sbohrer-EgGFQ3RFNTIP7C3xziwOQw@public.gmane.org> wrote:
> > > Is there any estimate on when we might see something like this upstream?
> > 
> > Could you elaborate a little on your use case for multicast IBoE
> > traffic? e.g how the setup looks like and how are your Ethernet
> > switches act to route that traffic.
> 
> I'm not sure exactly what you are asking here.  We do what I would
> imagine is a typical one to many UD multicast.  We code directly to
> libibvers and librdmacm, and everything is sent IBoE.
> 
> The hosts are in a spine/leaf configuration and all traffic is sent
> over vlans.  My understanding is that the multicast IBoE traffic is
> simply sent as broadcast and that the adapters do the necessary
> filtering.
> 
> Really from my point of view OFED already does what we want, but I
> would really like to see this supported upstream.

Is this something you are working on adding to upstream libibverbs?
I'd be happy to help with testing or whatever else might be needed.

Thanks,
Shawn

-- 

---------------------------------------------------------------
This email, along with any attachments, is confidential. If you 
believe you received this message in error, please contact the 
sender immediately and delete all copies of the message.  
Thank you.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] Add multicast IBoE support

[Shawn Bohrer]

Add multicast support for IBoE to the address handle creation flow.
Derived from work by Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>

Signed-off-by: Shawn Bohrer <sbohrer-EgGFQ3RFNTIP7C3xziwOQw@public.gmane.org>
---
 src/verbs.c |   38 +++++++++++++++++++++++++++++++-------
 1 files changed, 31 insertions(+), 7 deletions(-)

diff --git a/src/verbs.c b/src/verbs.c
index 199d107..2772de5 100644
--- a/src/verbs.c
+++ b/src/verbs.c
@@ -624,6 +624,11 @@ static int link_local_gid(const union ibv_gid *gid)
 	return 0;
 }
 
+static int is_multicast_gid(const union ibv_gid *gid)
+{
+	return gid->raw[0] == 0xff;
+}
+
 static uint16_t get_vlan_id(union ibv_gid *gid)
 {
 	uint16_t vid;
@@ -631,9 +636,12 @@ static uint16_t get_vlan_id(union ibv_gid *gid)
 	return vid < 0x1000 ? vid : 0xffff;
 }
 
-static int mlx4_resolve_grh_to_l2(struct mlx4_ah *ah, struct ibv_ah_attr *attr)
+static int mlx4_resolve_grh_to_l2(struct ibv_pd *pd, struct mlx4_ah *ah,
+				  struct ibv_ah_attr *attr)
 {
+	int err, i;
 	uint16_t vid;
+	union ibv_gid sgid;
 
 	if (link_local_gid(&attr->grh.dgid)) {
 		memcpy(ah->mac, &attr->grh.dgid.raw[8], 3);
@@ -641,13 +649,29 @@ static int mlx4_resolve_grh_to_l2(struct mlx4_ah *ah, struct ibv_ah_attr *attr)
 		ah->mac[0] ^= 2;
 
 		vid = get_vlan_id(&attr->grh.dgid);
-		if (vid != 0xffff) {
-			ah->av.port_pd |= htonl(1 << 29);
-			ah->vlan = vid | ((attr->sl & 7) << 13);
-		}
-		return 0;
+	} else if (is_multicast_gid(&attr->grh.dgid)) {
+		ah->mac[0] = 0x33;
+		ah->mac[1] = 0x33;
+		for (i = 2; i < 6; ++i)
+			ah->mac[i] = attr->grh.dgid.raw[i + 10];
+
+		err = ibv_query_gid(pd->context, attr->port_num,
+				    attr->grh.sgid_index, &sgid);
+		if (err)
+			return err;
+
+		ah->av.dlid = htons(0xc0000);
+		ah->av.port_pd |= htonl(1 << 31);
+
+		vid = get_vlan_id(&sgid);
 	} else
 		return 1;
+
+	if (vid != 0xFFFF) {
+		ah->av.port_pd |= htonl(1 << 29);
+		ah->vlan = vid | ((attr->sl & 7) << 13);
+	}
+	return 0;
 }
 
 struct ibv_ah *mlx4_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr)
@@ -687,7 +711,7 @@ struct ibv_ah *mlx4_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr)
 	}
 
 	if (port_attr.link_layer == IBV_LINK_LAYER_ETHERNET)
-		if (mlx4_resolve_grh_to_l2(ah, attr)) {
+		if (mlx4_resolve_grh_to_l2(pd, ah, attr)) {
 			free(ah);
 			return NULL;
 		}
-- 
1.7.7.5


-- 

---------------------------------------------------------------
This email, along with any attachments, is confidential. If you 
believe you received this message in error, please contact the 
sender immediately and delete all copies of the message.  
Thank you.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Add multicast IBoE support

[Shawn Bohrer]

Sorry, probably should have mentioned this is a patch for libmlx4.

On Wed, Jan 25, 2012 at 04:56:48PM -0600, Shawn Bohrer wrote:
> Add multicast support for IBoE to the address handle creation flow.
> Derived from work by Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
> 
> Signed-off-by: Shawn Bohrer <sbohrer-EgGFQ3RFNTIP7C3xziwOQw@public.gmane.org>
> ---
>  src/verbs.c |   38 +++++++++++++++++++++++++++++++-------
>  1 files changed, 31 insertions(+), 7 deletions(-)
> 
> diff --git a/src/verbs.c b/src/verbs.c
> index 199d107..2772de5 100644
> --- a/src/verbs.c
> +++ b/src/verbs.c
> @@ -624,6 +624,11 @@ static int link_local_gid(const union ibv_gid *gid)
>  	return 0;
>  }
>  
> +static int is_multicast_gid(const union ibv_gid *gid)
> +{
> +	return gid->raw[0] == 0xff;
> +}
> +
>  static uint16_t get_vlan_id(union ibv_gid *gid)
>  {
>  	uint16_t vid;
> @@ -631,9 +636,12 @@ static uint16_t get_vlan_id(union ibv_gid *gid)
>  	return vid < 0x1000 ? vid : 0xffff;
>  }
>  
> -static int mlx4_resolve_grh_to_l2(struct mlx4_ah *ah, struct ibv_ah_attr *attr)
> +static int mlx4_resolve_grh_to_l2(struct ibv_pd *pd, struct mlx4_ah *ah,
> +				  struct ibv_ah_attr *attr)
>  {
> +	int err, i;
>  	uint16_t vid;
> +	union ibv_gid sgid;
>  
>  	if (link_local_gid(&attr->grh.dgid)) {
>  		memcpy(ah->mac, &attr->grh.dgid.raw[8], 3);
> @@ -641,13 +649,29 @@ static int mlx4_resolve_grh_to_l2(struct mlx4_ah *ah, struct ibv_ah_attr *attr)
>  		ah->mac[0] ^= 2;
>  
>  		vid = get_vlan_id(&attr->grh.dgid);
> -		if (vid != 0xffff) {
> -			ah->av.port_pd |= htonl(1 << 29);
> -			ah->vlan = vid | ((attr->sl & 7) << 13);
> -		}
> -		return 0;
> +	} else if (is_multicast_gid(&attr->grh.dgid)) {
> +		ah->mac[0] = 0x33;
> +		ah->mac[1] = 0x33;
> +		for (i = 2; i < 6; ++i)
> +			ah->mac[i] = attr->grh.dgid.raw[i + 10];
> +
> +		err = ibv_query_gid(pd->context, attr->port_num,
> +				    attr->grh.sgid_index, &sgid);
> +		if (err)
> +			return err;
> +
> +		ah->av.dlid = htons(0xc0000);
> +		ah->av.port_pd |= htonl(1 << 31);
> +
> +		vid = get_vlan_id(&sgid);
>  	} else
>  		return 1;
> +
> +	if (vid != 0xFFFF) {
> +		ah->av.port_pd |= htonl(1 << 29);
> +		ah->vlan = vid | ((attr->sl & 7) << 13);
> +	}
> +	return 0;
>  }
>  
>  struct ibv_ah *mlx4_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr)
> @@ -687,7 +711,7 @@ struct ibv_ah *mlx4_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr)
>  	}
>  
>  	if (port_attr.link_layer == IBV_LINK_LAYER_ETHERNET)
> -		if (mlx4_resolve_grh_to_l2(ah, attr)) {
> +		if (mlx4_resolve_grh_to_l2(pd, ah, attr)) {
>  			free(ah);
>  			return NULL;
>  		}
> -- 
> 1.7.7.5
> 

-- 

---------------------------------------------------------------
This email, along with any attachments, is confidential. If you 
believe you received this message in error, please contact the 
sender immediately and delete all copies of the message.  
Thank you.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html