[PATCH 0/7] IBoE user space support

[PATCH 0/7] IBoE user space support

[Or Gerlitz]

Roland, the enclosed patch series enhances the user space IB
stack to support IBoE. This work is based on earlier patches
done by Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org> and posted during
August/2010 see some pointers below.

One notable change from the previous post, is keeping the UD related
resolution between GID to Ethernet L2 address objects (e.g mac, vlan)
within libmlx4. Another patch with notable change is the last, which
aligns libmlx4 with the kernel w.r.t supported devices.

[PATCH 1/7] libibverbs: Add link layer field port attribute
[PATCH 2/7] libibverbs: change kernel API to accept link layer
[PATCH 3/7] libibverbs: add GID change event
[PATCH 4/7] libibverbs: update examples for IBoE

[PATCH 5/7] libmlx4: add IBoE support
[PATCH 6/7] libmlx4: add IBoE UD/VLANs support

[PATCH 7/7] libmlx4: align the list of ConnectX devices supported with the kernel

Or.

libibverbs V10 from August 2010
http://marc.info/?l=linux-rdma&m=128283236815520
http://marc.info/?l=linux-rdma&m=128283237715545
http://marc.info/?l=linux-rdma&m=128283238115562
http://marc.info/?l=linux-rdma&m=128283239215578

libmlx4 V10 from August 2010
http://marc.info/?l=linux-rdma&m=128283239515584
http://marc.info/?l=linux-rdma&m=128283240415602
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 1/7] libibverbs: Add link layer field port attribute

[Or Gerlitz]

The new field has one of three values - IBV_LINK_LAYER_UNSPECIFIED,
IBV_LINK_LAYER_INFINIBAND, IBV_LINK_LAYER_ETHERNET. It can be used
by applications to know the link layer used by the port, which
can be either Infiniband or Ethernet.

The addition of the new field does not change the size of struct
ibv_port_attr due to alignment of the preceding fields. As such
binary compatibility between the library to applications is kept,
since old apps running over new library do not read this field,
and new apps running over old library will determine the link
layer as unspecified and hence take their IB code path.

The solution was suggested by: Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org>
and Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>

Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
Signed-off-by: Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
---
 include/infiniband/verbs.h |   21 +++++++++++++++++++++
 man/ibv_query_port.3       |    4 ++++
 2 files changed, 25 insertions(+), 0 deletions(-)

diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index 0f1cb2e..17df3ff 100644
--- a/include/infiniband/verbs.h
+++ b/include/infiniband/verbs.h
@@ -161,6 +161,12 @@ enum ibv_port_state {
 	IBV_PORT_ACTIVE_DEFER	= 5
 };

+enum {
+	IBV_LINK_LAYER_UNSPECIFIED,
+	IBV_LINK_LAYER_INFINIBAND,
+	IBV_LINK_LAYER_ETHERNET,
+};
+
 struct ibv_port_attr {
 	enum ibv_port_state	state;
 	enum ibv_mtu		max_mtu;
@@ -181,6 +187,8 @@ struct ibv_port_attr {
 	uint8_t			active_width;
 	uint8_t			active_speed;
 	uint8_t			phys_state;
+	uint8_t			link_layer;
+	uint8_t			pad;
 };

 enum ibv_event_type {
@@ -693,6 +701,16 @@ struct ibv_context {
 	void		       *abi_compat;
 };

+static inline int ___ibv_query_port(struct ibv_context *context,
+				    uint8_t port_num,
+				    struct ibv_port_attr *port_attr)
+{
+	port_attr->link_layer = IBV_LINK_LAYER_UNSPECIFIED;
+	port_attr->pad = 0;
+
+	return context->ops.query_port(context, port_num, port_attr);
+}
+
 /**
  * ibv_get_device_list - Get list of IB devices currently available
  * @num_devices: optional.  if non-NULL, set to the number of devices
@@ -1097,4 +1115,7 @@ END_C_DECLS

 #  undef __attribute_const

+#define ibv_query_port(context, port_num, port_attr) \
+	___ibv_query_port(context, port_num, port_attr)
+
 #endif /* INFINIBAND_VERBS_H */
diff --git a/man/ibv_query_port.3 b/man/ibv_query_port.3
index 882470d..9bedd90 100644
--- a/man/ibv_query_port.3
+++ b/man/ibv_query_port.3
@@ -44,9 +44,13 @@ uint8_t                 init_type_reply;/* Type of initialization performed by S
 uint8_t                 active_width;   /* Currently active link width */
 uint8_t                 active_speed;   /* Currently active link speed */
 uint8_t                 phys_state;     /* Physical port state */
+uint8_t                 link_layer;     /* link layer protocol of the port */
 .in -8
 };
 .sp
+possible values for the link layer field are IBV_LINK_LAYER_INFINIBAND,
+IBV_LINK_LAYER_ETHERNET, or IBV_LINK_LAYER_UNSPECIFIED.
+.sp
 .fi
 .SH "RETURN VALUE"
 .B ibv_query_port()
-- 
1.5.5

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 2/7] libibverbs: change kernel API to accept link layer

[Or Gerlitz]

Modify the code to allow passing the link layer of a port from the
kernel to the library.

The new field does not change the size of struct ibv_query_port_resp
as it replaces a reserved field. As such binary compatibility between
the kernel to the library is kept, since old kernel running below new
library will not touch that field so it will be resolved as unspecified,
and old library running over new kernel will ignore the setting done by
the kernel.

The solution was suggested by: Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org>
and Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>

Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
Signed-off-by: Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
---
 include/infiniband/kern-abi.h |    3 ++-
 src/cmd.c                     |    1 +
 2 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/include/infiniband/kern-abi.h b/include/infiniband/kern-abi.h
index 0db083a..619ea7e 100644
--- a/include/infiniband/kern-abi.h
+++ b/include/infiniband/kern-abi.h
@@ -223,7 +223,8 @@ struct ibv_query_port_resp {
 	__u8  active_width;
 	__u8  active_speed;
 	__u8  phys_state;
-	__u8  reserved[3];
+	__u8  link_layer;
+	__u8  reserved[2];
 };

 struct ibv_alloc_pd {
diff --git a/src/cmd.c b/src/cmd.c
index cbd5288..39af833 100644
--- a/src/cmd.c
+++ b/src/cmd.c
@@ -196,6 +196,7 @@ int ibv_cmd_query_port(struct ibv_context *context, uint8_t port_num,
 	port_attr->active_width    = resp.active_width;
 	port_attr->active_speed    = resp.active_speed;
 	port_attr->phys_state      = resp.phys_state;
+	port_attr->link_layer      = resp.link_layer;

 	return 0;
 }
-- 
1.5.5


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 3/7] libibverbs: add GID change event

[Or Gerlitz]

Add IB GID change event which is generated by the kernel
IBoE stack when the HW driver updates the GID table.

Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
Signed-off-by: Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
---
 examples/asyncwatch.c      |    2 ++
 include/infiniband/verbs.h |    3 ++-
 man/ibv_get_async_event.3  |    2 ++
 3 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/examples/asyncwatch.c b/examples/asyncwatch.c
index 5510a29..da7ebd4 100644
--- a/examples/asyncwatch.c
+++ b/examples/asyncwatch.c
@@ -57,6 +57,8 @@ static const char *event_name_str(enum ibv_event_type event_type)
 		return "IBV_EVENT_SM_CHANGE";
 	case IBV_EVENT_CLIENT_REREGISTER:
 		return "IBV_EVENT_CLIENT_REREGISTER";
+	case IBV_EVENT_GID_CHANGE:
+		return "IBV_EVENT_GID_CHANGE";

 	case IBV_EVENT_CQ_ERR:
 	case IBV_EVENT_QP_FATAL:
diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index 17df3ff..0dd79f6 100644
--- a/include/infiniband/verbs.h
+++ b/include/infiniband/verbs.h
@@ -209,7 +209,8 @@ enum ibv_event_type {
 	IBV_EVENT_SRQ_ERR,
 	IBV_EVENT_SRQ_LIMIT_REACHED,
 	IBV_EVENT_QP_LAST_WQE_REACHED,
-	IBV_EVENT_CLIENT_REREGISTER
+	IBV_EVENT_CLIENT_REREGISTER,
+	IBV_EVENT_GID_CHANGE,
 };

 struct ibv_async_event {
diff --git a/man/ibv_get_async_event.3 b/man/ibv_get_async_event.3
index acb6257..a76dc0c 100644
--- a/man/ibv_get_async_event.3
+++ b/man/ibv_get_async_event.3
@@ -81,6 +81,8 @@ following events:
 .B IBV_EVENT_SM_CHANGE \fR SM was changed on a port
 .TP
 .B IBV_EVENT_CLIENT_REREGISTER \fR SM sent a CLIENT_REREGISTER request to a port
+.TP
+.B IBV_EVENT_GID_CHANGE \fR GID table was changed on a port
 .PP
 .I CA events:
 .TP
-- 
1.5.5


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 4/7] libibverbs: Update examples for IBoE

[Or Gerlitz]

Since IBoE requires usage of GRH, update ibv_*_pinpong examples to accept
GIDs. GIDs are given as an index to the local port's table and are exchanged
between the client and the server through the socket connection.

Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
Signed-off-by: Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
---
 examples/devinfo.c      |   15 +++++++
 examples/pingpong.c     |   32 +++++++++++++++
 examples/pingpong.h     |    4 ++
 examples/rc_pingpong.c  |   97 ++++++++++++++++++++++++++++++++++++-----------
 examples/srq_pingpong.c |   96 +++++++++++++++++++++++++++++++++++-----------
 examples/uc_pingpong.c  |   97 ++++++++++++++++++++++++++++++++++++-----------
 examples/ud_pingpong.c  |   89 ++++++++++++++++++++++++++++++++-----------
 7 files changed, 342 insertions(+), 88 deletions(-)

diff --git a/examples/devinfo.c b/examples/devinfo.c
index 84f95c7..9c472a0 100644
--- a/examples/devinfo.c
+++ b/examples/devinfo.c
@@ -184,6 +184,19 @@ static int print_all_port_gids(struct ibv_context *ctx, uint8_t port_num, int tb
 	return rc;
 }

+static const char *link_layer_str(uint8_t link_layer)
+{
+	switch (link_layer) {
+	case IBV_LINK_LAYER_UNSPECIFIED:
+	case IBV_LINK_LAYER_INFINIBAND:
+		return "IB";
+	case IBV_LINK_LAYER_ETHERNET:
+		return "Ethernet";
+	default:
+		return "Unknown";
+	}
+}
+
 static int print_hca_cap(struct ibv_device *ib_dev, uint8_t ib_port)
 {
 	struct ibv_context *ctx;
@@ -284,6 +297,8 @@ static int print_hca_cap(struct ibv_device *ib_dev, uint8_t ib_port)
 		printf("\t\t\tsm_lid:\t\t\t%d\n", port_attr.sm_lid);
 		printf("\t\t\tport_lid:\t\t%d\n", port_attr.lid);
 		printf("\t\t\tport_lmc:\t\t0x%02x\n", port_attr.lmc);
+		printf("\t\t\tlink_layer:\t\t%s\n",
+					link_layer_str(port_attr.link_layer));

 		if (verbose) {
 			printf("\t\t\tmax_msg_sz:\t\t0x%x\n", port_attr.max_msg_sz);
diff --git a/examples/pingpong.c b/examples/pingpong.c
index b916f59..d06ba84 100644
--- a/examples/pingpong.c
+++ b/examples/pingpong.c
@@ -31,6 +31,10 @@
  */

 #include "pingpong.h"
+#include <arpa/inet.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>

 enum ibv_mtu pp_mtu_to_enum(int mtu)
 {
@@ -53,3 +57,31 @@ uint16_t pp_get_local_lid(struct ibv_context *context, int port)

 	return attr.lid;
 }
+
+int pp_get_port_info(struct ibv_context *context, int port,
+		     struct ibv_port_attr *attr)
+{
+	return ibv_query_port(context, port, attr);
+}
+
+void wire_gid_to_gid(const char *wgid, union ibv_gid *gid)
+{
+	char tmp[9];
+	uint32_t v32;
+	int i;
+
+	for (tmp[8] = 0, i = 0; i < 4; ++i) {
+		memcpy(tmp, wgid + i * 8, 8);
+		sscanf(tmp, "%x", &v32);
+		*(uint32_t *)(&gid->raw[i * 4]) = ntohl(v32);
+	}
+}
+
+void gid_to_wire_gid(const union ibv_gid *gid, char wgid[])
+{
+	int i;
+
+	for (i = 0; i < 4; ++i)
+		sprintf(&wgid[i * 8], "%08x",
+				htonl(*(uint32_t *)(gid->raw + i * 4)));
+}
diff --git a/examples/pingpong.h b/examples/pingpong.h
index 71d7c3f..9cdc03e 100644
--- a/examples/pingpong.h
+++ b/examples/pingpong.h
@@ -37,5 +37,9 @@

 enum ibv_mtu pp_mtu_to_enum(int mtu);
 uint16_t pp_get_local_lid(struct ibv_context *context, int port);
+int pp_get_port_info(struct ibv_context *context, int port,
+		     struct ibv_port_attr *attr);
+void wire_gid_to_gid(const char *wgid, union ibv_gid *gid);
+void gid_to_wire_gid(const union ibv_gid *gid, char wgid[]);

 #endif /* IBV_PINGPONG_H */
diff --git a/examples/rc_pingpong.c b/examples/rc_pingpong.c
index fa969e0..0b7f3e0 100644
--- a/examples/rc_pingpong.c
+++ b/examples/rc_pingpong.c
@@ -67,17 +67,19 @@ struct pingpong_context {
 	int			 size;
 	int			 rx_depth;
 	int			 pending;
+	struct ibv_port_attr     portinfo;
 };

 struct pingpong_dest {
 	int lid;
 	int qpn;
 	int psn;
+	union ibv_gid gid;
 };

 static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn,
 			  enum ibv_mtu mtu, int sl,
-			  struct pingpong_dest *dest)
+			  struct pingpong_dest *dest, int sgid_idx)
 {
 	struct ibv_qp_attr attr = {
 		.qp_state		= IBV_QPS_RTR,
@@ -94,6 +96,13 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn,
 			.port_num	= port
 		}
 	};
+
+	if (dest->gid.global.interface_id) {
+		attr.ah_attr.is_global = 1;
+		attr.ah_attr.grh.hop_limit = 1;
+		attr.ah_attr.grh.dgid = dest->gid;
+		attr.ah_attr.grh.sgid_index = sgid_idx;
+	}
 	if (ibv_modify_qp(ctx->qp, &attr,
 			  IBV_QP_STATE              |
 			  IBV_QP_AV                 |
@@ -135,10 +144,11 @@ static struct pingpong_dest *pp_client_exch_dest(const char *servername, int por
 		.ai_socktype = SOCK_STREAM
 	};
 	char *service;
-	char msg[sizeof "0000:000000:000000"];
+	char msg[sizeof "0000:000000:000000:00000000000000000000000000000000"];
 	int n;
 	int sockfd = -1;
 	struct pingpong_dest *rem_dest = NULL;
+	char gid[33];

 	if (asprintf(&service, "%d", port) < 0)
 		return NULL;
@@ -169,7 +179,9 @@ static struct pingpong_dest *pp_client_exch_dest(const char *servername, int por
 		return NULL;
 	}

-	sprintf(msg, "%04x:%06x:%06x", my_dest->lid, my_dest->qpn, my_dest->psn);
+	gid_to_wire_gid(&my_dest->gid, gid);
+	sprintf(msg, "%04x:%06x:%06x:%s", my_dest->lid, my_dest->qpn,
+							my_dest->psn, gid);
 	if (write(sockfd, msg, sizeof msg) != sizeof msg) {
 		fprintf(stderr, "Couldn't send local address\n");
 		goto out;
@@ -187,7 +199,9 @@ static struct pingpong_dest *pp_client_exch_dest(const char *servername, int por
 	if (!rem_dest)
 		goto out;

-	sscanf(msg, "%x:%x:%x", &rem_dest->lid, &rem_dest->qpn, &rem_dest->psn);
+	sscanf(msg, "%x:%x:%x:%s", &rem_dest->lid, &rem_dest->qpn,
+						&rem_dest->psn, gid);
+	wire_gid_to_gid(gid, &rem_dest->gid);

 out:
 	close(sockfd);
@@ -197,7 +211,8 @@ out:
 static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 						 int ib_port, enum ibv_mtu mtu,
 						 int port, int sl,
-						 const struct pingpong_dest *my_dest)
+						 const struct pingpong_dest *my_dest,
+						 int sgid_idx)
 {
 	struct addrinfo *res, *t;
 	struct addrinfo hints = {
@@ -206,10 +221,11 @@ static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 		.ai_socktype = SOCK_STREAM
 	};
 	char *service;
-	char msg[sizeof "0000:000000:000000"];
+	char msg[sizeof "0000:000000:000000:00000000000000000000000000000000"];
 	int n;
 	int sockfd = -1, connfd;
 	struct pingpong_dest *rem_dest = NULL;
+	char gid[33];

 	if (asprintf(&service, "%d", port) < 0)
 		return NULL;
@@ -263,16 +279,22 @@ static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 	if (!rem_dest)
 		goto out;

-	sscanf(msg, "%x:%x:%x", &rem_dest->lid, &rem_dest->qpn, &rem_dest->psn);
+	sscanf(msg, "%x:%x:%x:%s", &rem_dest->lid, &rem_dest->qpn,
+							&rem_dest->psn, gid);
+	wire_gid_to_gid(gid, &rem_dest->gid);

-	if (pp_connect_ctx(ctx, ib_port, my_dest->psn, mtu, sl, rem_dest)) {
+	if (pp_connect_ctx(ctx, ib_port, my_dest->psn, mtu, sl, rem_dest,
+								sgid_idx)) {
 		fprintf(stderr, "Couldn't connect to remote QP\n");
 		free(rem_dest);
 		rem_dest = NULL;
 		goto out;
 	}

-	sprintf(msg, "%04x:%06x:%06x", my_dest->lid, my_dest->qpn, my_dest->psn);
+
+	gid_to_wire_gid(&my_dest->gid, gid);
+	sprintf(msg, "%04x:%06x:%06x:%s", my_dest->lid, my_dest->qpn,
+							my_dest->psn, gid);
 	if (write(connfd, msg, sizeof msg) != sizeof msg) {
 		fprintf(stderr, "Couldn't send local address\n");
 		free(rem_dest);
@@ -293,7 +315,7 @@ static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev, int size,
 {
 	struct pingpong_context *ctx;

-	ctx = malloc(sizeof *ctx);
+	ctx = calloc(1, sizeof *ctx);
 	if (!ctx)
 		return NULL;

@@ -306,7 +328,8 @@ static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev, int size,
 		return NULL;
 	}

-	memset(ctx->buf, 0, size);
+	/* FIXME memset(ctx->buf, 0, size); */
+	memset(ctx->buf, 0x7b, size);

 	ctx->context = ibv_open_device(ib_dev);
 	if (!ctx->context) {
@@ -481,6 +504,7 @@ static void usage(const char *argv0)
 	printf("  -n, --iters=<iters>    number of exchanges (default 1000)\n");
 	printf("  -l, --sl=<sl>          service level value\n");
 	printf("  -e, --events           sleep on CQ events (default poll)\n");
+	printf("  -g, --gid-idx=<gid index> local port gid index\n");
 }

 int main(int argc, char *argv[])
@@ -504,6 +528,8 @@ int main(int argc, char *argv[])
 	int                      rcnt, scnt;
 	int                      num_cq_events = 0;
 	int                      sl = 0;
+	int			 gidx = -1;
+	char			 gid[33];

 	srand48(getpid() * time(NULL));

@@ -520,10 +546,12 @@ int main(int argc, char *argv[])
 			{ .name = "iters",    .has_arg = 1, .val = 'n' },
 			{ .name = "sl",       .has_arg = 1, .val = 'l' },
 			{ .name = "events",   .has_arg = 0, .val = 'e' },
+			{ .name = "gid-idx",  .has_arg = 1, .val = 'g' },
 			{ 0 }
 		};

-		c = getopt_long(argc, argv, "p:d:i:s:m:r:n:l:e", long_options, NULL);
+		c = getopt_long(argc, argv, "p:d:i:s:m:r:n:l:eg:",
+							long_options, NULL);
 		if (c == -1)
 			break;

@@ -576,6 +604,10 @@ int main(int argc, char *argv[])
 			++use_event;
 			break;

+		case 'g':
+			gidx = strtol(optarg, NULL, 0);
+			break;
+
 		default:
 			usage(argv[0]);
 			return 1;
@@ -631,30 +663,50 @@ int main(int argc, char *argv[])
 			return 1;
 		}

-	my_dest.lid = pp_get_local_lid(ctx->context, ib_port);
-	my_dest.qpn = ctx->qp->qp_num;
-	my_dest.psn = lrand48() & 0xffffff;
-	if (!my_dest.lid) {
+
+	if (pp_get_port_info(ctx->context, ib_port, &ctx->portinfo)) {
+		fprintf(stderr, "Couldn't get port info\n");
+		return 1;
+	}
+
+	my_dest.lid = ctx->portinfo.lid;
+	if (ctx->portinfo.link_layer != IBV_LINK_LAYER_ETHERNET &&
+							!my_dest.lid) {
 		fprintf(stderr, "Couldn't get local LID\n");
 		return 1;
 	}

-	printf("  local address:  LID 0x%04x, QPN 0x%06x, PSN 0x%06x\n",
-	       my_dest.lid, my_dest.qpn, my_dest.psn);
+	if (gidx >= 0) {
+		if (ibv_query_gid(ctx->context, ib_port, gidx, &my_dest.gid)) {
+			fprintf(stderr, "can't read sgid of index %d\n", gidx);
+			return 1;
+		}
+	} else
+		memset(&my_dest.gid, 0, sizeof my_dest.gid);
+
+	my_dest.qpn = ctx->qp->qp_num;
+	my_dest.psn = lrand48() & 0xffffff;
+	inet_ntop(AF_INET6, &my_dest.gid, gid, sizeof gid);
+	printf("  local address:  LID 0x%04x, QPN 0x%06x, PSN 0x%06x, GID %s\n",
+	       my_dest.lid, my_dest.qpn, my_dest.psn, gid);
+

 	if (servername)
 		rem_dest = pp_client_exch_dest(servername, port, &my_dest);
 	else
-		rem_dest = pp_server_exch_dest(ctx, ib_port, mtu, port, sl, &my_dest);
+		rem_dest = pp_server_exch_dest(ctx, ib_port, mtu, port, sl,
+								&my_dest, gidx);

 	if (!rem_dest)
 		return 1;

-	printf("  remote address: LID 0x%04x, QPN 0x%06x, PSN 0x%06x\n",
-	       rem_dest->lid, rem_dest->qpn, rem_dest->psn);
+	inet_ntop(AF_INET6, &rem_dest->gid, gid, sizeof gid);
+	printf("  remote address: LID 0x%04x, QPN 0x%06x, PSN 0x%06x, GID %s\n",
+	       rem_dest->lid, rem_dest->qpn, rem_dest->psn, gid);

 	if (servername)
-		if (pp_connect_ctx(ctx, ib_port, my_dest.psn, mtu, sl, rem_dest))
+		if (pp_connect_ctx(ctx, ib_port, my_dest.psn, mtu, sl, rem_dest,
+					gidx))
 			return 1;

 	ctx->pending = PINGPONG_RECV_WRID;
@@ -706,6 +758,7 @@ int main(int argc, char *argv[])
 					fprintf(stderr, "poll CQ failed %d\n", ne);
 					return 1;
 				}
+
 			} while (!use_event && ne < 1);

 			for (i = 0; i < ne; ++i) {
diff --git a/examples/srq_pingpong.c b/examples/srq_pingpong.c
index 1e36c57..298dca4 100644
--- a/examples/srq_pingpong.c
+++ b/examples/srq_pingpong.c
@@ -71,17 +71,19 @@ struct pingpong_context {
 	int			 num_qp;
 	int			 rx_depth;
 	int			 pending[MAX_QP];
+	struct ibv_port_attr	 portinfo;
 };

 struct pingpong_dest {
 	int lid;
 	int qpn;
 	int psn;
+	union ibv_gid gid;
 };

 static int pp_connect_ctx(struct pingpong_context *ctx, int port, enum ibv_mtu mtu,
 			  int sl, const struct pingpong_dest *my_dest,
-			  const struct pingpong_dest *dest)
+			  const struct pingpong_dest *dest, int sgid_idx)
 {
 	int i;

@@ -101,6 +103,13 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, enum ibv_mtu m
 				.port_num	= port
 			}
 		};
+
+		if (dest->gid.global.interface_id) {
+			attr.ah_attr.is_global = 1;
+			attr.ah_attr.grh.hop_limit = 1;
+			attr.ah_attr.grh.dgid = dest->gid;
+			attr.ah_attr.grh.sgid_index = sgid_idx;
+		}
 		if (ibv_modify_qp(ctx->qp[i], &attr,
 				  IBV_QP_STATE              |
 				  IBV_QP_AV                 |
@@ -143,12 +152,13 @@ static struct pingpong_dest *pp_client_exch_dest(const char *servername, int por
 		.ai_socktype = SOCK_STREAM
 	};
 	char *service;
-	char msg[sizeof "0000:000000:000000"];
+	char msg[sizeof "0000:000000:000000:00000000000000000000000000000000"];
 	int n;
 	int r;
 	int i;
 	int sockfd = -1;
 	struct pingpong_dest *rem_dest = NULL;
+	char gid[33];

 	if (asprintf(&service, "%d", port) < 0)
 		return NULL;
@@ -180,7 +190,9 @@ static struct pingpong_dest *pp_client_exch_dest(const char *servername, int por
 	}

 	for (i = 0; i < MAX_QP; ++i) {
-		sprintf(msg, "%04x:%06x:%06x", my_dest[i].lid, my_dest[i].qpn, my_dest[i].psn);
+		gid_to_wire_gid(&my_dest[i].gid, gid);
+		sprintf(msg, "%04x:%06x:%06x:%s", my_dest[i].lid,
+					my_dest[i].qpn, my_dest[i].psn, gid);
 		if (write(sockfd, msg, sizeof msg) != sizeof msg) {
 			fprintf(stderr, "Couldn't send local address\n");
 			goto out;
@@ -204,8 +216,9 @@ static struct pingpong_dest *pp_client_exch_dest(const char *servername, int por
 			n += r;
 		}

-		sscanf(msg, "%x:%x:%x",
-		       &rem_dest[i].lid, &rem_dest[i].qpn, &rem_dest[i].psn);
+		sscanf(msg, "%x:%x:%x:%s", &rem_dest[i].lid, &rem_dest[i].qpn,
+							&rem_dest[i].psn, gid);
+		wire_gid_to_gid(gid, &rem_dest[i].gid);
 	}

 	write(sockfd, "done", sizeof "done");
@@ -218,7 +231,8 @@ out:
 static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 						 int ib_port, enum ibv_mtu mtu,
 						 int port, int sl,
-						 const struct pingpong_dest *my_dest)
+						 const struct pingpong_dest *my_dest,
+						 int sgid_idx)
 {
 	struct addrinfo *res, *t;
 	struct addrinfo hints = {
@@ -227,12 +241,13 @@ static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 		.ai_socktype = SOCK_STREAM
 	};
 	char *service;
-	char msg[sizeof "0000:000000:000000"];
+	char msg[sizeof "0000:000000:000000:00000000000000000000000000000000"];
 	int n;
 	int r;
 	int i;
 	int sockfd = -1, connfd;
 	struct pingpong_dest *rem_dest = NULL;
+	char gid[33];

 	if (asprintf(&service, "%d", port) < 0)
 		return NULL;
@@ -292,11 +307,13 @@ static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 			n += r;
 		}

-		sscanf(msg, "%x:%x:%x",
-		       &rem_dest[i].lid, &rem_dest[i].qpn, &rem_dest[i].psn);
+		sscanf(msg, "%x:%x:%x:%s", &rem_dest[i].lid, &rem_dest[i].qpn,
+							&rem_dest[i].psn, gid);
+		wire_gid_to_gid(gid, &rem_dest[i].gid);
 	}

-	if (pp_connect_ctx(ctx, ib_port, mtu, sl, my_dest, rem_dest)) {
+	if (pp_connect_ctx(ctx, ib_port, mtu, sl, my_dest, rem_dest,
+								sgid_idx)) {
 		fprintf(stderr, "Couldn't connect to remote QP\n");
 		free(rem_dest);
 		rem_dest = NULL;
@@ -304,7 +321,9 @@ static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 	}

 	for (i = 0; i < MAX_QP; ++i) {
-		sprintf(msg, "%04x:%06x:%06x", my_dest[i].lid, my_dest[i].qpn, my_dest[i].psn);
+		gid_to_wire_gid(&my_dest[i].gid, gid);
+		sprintf(msg, "%04x:%06x:%06x:%s", my_dest[i].lid,
+					my_dest[i].qpn, my_dest[i].psn, gid);
 		if (write(connfd, msg, sizeof msg) != sizeof msg) {
 			fprintf(stderr, "Couldn't send local address\n");
 			free(rem_dest);
@@ -327,7 +346,7 @@ static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev, int size,
 	struct pingpong_context *ctx;
 	int i;

-	ctx = malloc(sizeof *ctx);
+	ctx = calloc(1, sizeof *ctx);
 	if (!ctx)
 		return NULL;

@@ -551,6 +570,7 @@ static void usage(const char *argv0)
 	printf("  -n, --iters=<iters>    number of exchanges per QP(default 1000)\n");
 	printf("  -l, --sl=<sl>          service level value\n");
 	printf("  -e, --events           sleep on CQ events (default poll)\n");
+	printf("  -g, --gid-idx=<gid index> local port gid index\n");
 }

 int main(int argc, char *argv[])
@@ -578,6 +598,8 @@ int main(int argc, char *argv[])
 	int                      i;
 	int                      num_cq_events = 0;
 	int                      sl = 0;
+	int			 gidx = -1;
+	char			 gid[33];

 	srand48(getpid() * time(NULL));

@@ -595,10 +617,12 @@ int main(int argc, char *argv[])
 			{ .name = "iters",    .has_arg = 1, .val = 'n' },
 			{ .name = "sl",       .has_arg = 1, .val = 'l' },
 			{ .name = "events",   .has_arg = 0, .val = 'e' },
+			{ .name = "gid-idx",  .has_arg = 1, .val = 'g' },
 			{ 0 }
 		};

-		c = getopt_long(argc, argv, "p:d:i:s:m:q:r:n:l:e", long_options, NULL);
+		c = getopt_long(argc, argv, "p:d:i:s:m:q:r:n:l:eg:",
+							long_options, NULL);
 		if (c == -1)
 			break;

@@ -655,6 +679,10 @@ int main(int argc, char *argv[])
 			++use_event;
 			break;

+		case 'g':
+			gidx = strtol(optarg, NULL, 0);
+			break;
+
 		default:
 			usage(argv[0]);
 			return 1;
@@ -722,33 +750,57 @@ int main(int argc, char *argv[])

 	memset(my_dest, 0, sizeof my_dest);

+	if (pp_get_port_info(ctx->context, ib_port, &ctx->portinfo)) {
+		fprintf(stderr, "Couldn't get port info\n");
+		return 1;
+	}
 	for (i = 0; i < num_qp; ++i) {
 		my_dest[i].qpn = ctx->qp[i]->qp_num;
 		my_dest[i].psn = lrand48() & 0xffffff;
-		my_dest[i].lid = pp_get_local_lid(ctx->context, ib_port);
-		if (!my_dest[i].lid) {
+		my_dest[i].lid = ctx->portinfo.lid;
+		if (ctx->portinfo.link_layer != IBV_LINK_LAYER_ETHERNET
+							&& !my_dest[i].lid) {
 			fprintf(stderr, "Couldn't get local LID\n");
 			return 1;
 		}

-		printf("  local address:  LID 0x%04x, QPN 0x%06x, PSN 0x%06x\n",
-		       my_dest[i].lid, my_dest[i].qpn, my_dest[i].psn);
+		if (gidx >= 0) {
+			if (ibv_query_gid(ctx->context, ib_port, gidx,
+							&my_dest[i].gid)) {
+				fprintf(stderr, "Could not get local gid for "
+							"gid index %d\n", gidx);
+				return 1;
+			}
+		} else
+			memset(&my_dest[i].gid, 0, sizeof my_dest[i].gid);
+
+		inet_ntop(AF_INET6, &my_dest[i].gid, gid, sizeof gid);
+		printf("  local address:  LID 0x%04x, QPN 0x%06x, PSN 0x%06x, "
+			"GID %s\n", my_dest[i].lid, my_dest[i].qpn,
+			my_dest[i].psn, gid);
 	}

 	if (servername)
 		rem_dest = pp_client_exch_dest(servername, port, my_dest);
 	else
-		rem_dest = pp_server_exch_dest(ctx, ib_port, mtu, port, sl, my_dest);
+		rem_dest = pp_server_exch_dest(ctx, ib_port, mtu, port, sl,
+								my_dest, gidx);

 	if (!rem_dest)
 		return 1;

-	for (i = 0; i < num_qp; ++i)
-		printf("  remote address: LID 0x%04x, QPN 0x%06x, PSN 0x%06x\n",
-		       rem_dest[i].lid, rem_dest[i].qpn, rem_dest[i].psn);
+	inet_ntop(AF_INET6, &rem_dest->gid, gid, sizeof gid);
+
+	for (i = 0; i < num_qp; ++i) {
+		inet_ntop(AF_INET6, &rem_dest[i].gid, gid, sizeof gid);
+		printf("  remote address: LID 0x%04x, QPN 0x%06x, PSN 0x%06x, "
+			"GID %s\n", rem_dest[i].lid, rem_dest[i].qpn,
+			rem_dest[i].psn, gid);
+	}

 	if (servername)
-		if (pp_connect_ctx(ctx, ib_port, mtu, sl, my_dest, rem_dest))
+		if (pp_connect_ctx(ctx, ib_port, mtu, sl, my_dest, rem_dest,
+									gidx))
 			return 1;

 	if (servername)
diff --git a/examples/uc_pingpong.c b/examples/uc_pingpong.c
index 6f31247..4c3fa32 100644
--- a/examples/uc_pingpong.c
+++ b/examples/uc_pingpong.c
@@ -67,17 +67,19 @@ struct pingpong_context {
 	int			 size;
 	int			 rx_depth;
 	int			 pending;
+	struct ibv_port_attr     portinfo;
 };

 struct pingpong_dest {
 	int lid;
 	int qpn;
 	int psn;
+	union ibv_gid gid;
 };

 static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn,
 			  enum ibv_mtu mtu, int sl,
-			  struct pingpong_dest *dest)
+			  struct pingpong_dest *dest, int sgid_idx)
 {
 	struct ibv_qp_attr attr = {
 		.qp_state		= IBV_QPS_RTR,
@@ -92,6 +94,13 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn,
 			.port_num	= port
 		}
 	};
+
+	if (dest->gid.global.interface_id) {
+		attr.ah_attr.is_global = 1;
+		attr.ah_attr.grh.hop_limit = 1;
+		attr.ah_attr.grh.dgid = dest->gid;
+		attr.ah_attr.grh.sgid_index = sgid_idx;
+	}
 	if (ibv_modify_qp(ctx->qp, &attr,
 			  IBV_QP_STATE              |
 			  IBV_QP_AV                 |
@@ -123,10 +132,11 @@ static struct pingpong_dest *pp_client_exch_dest(const char *servername, int por
 		.ai_socktype = SOCK_STREAM
 	};
 	char *service;
-	char msg[sizeof "0000:000000:000000"];
+	char msg[sizeof "0000:000000:000000:00000000000000000000000000000000"];
 	int n;
 	int sockfd = -1;
 	struct pingpong_dest *rem_dest = NULL;
+	char gid[33];

 	if (asprintf(&service, "%d", port) < 0)
 		return NULL;
@@ -157,7 +167,9 @@ static struct pingpong_dest *pp_client_exch_dest(const char *servername, int por
 		return NULL;
 	}

-	sprintf(msg, "%04x:%06x:%06x", my_dest->lid, my_dest->qpn, my_dest->psn);
+	gid_to_wire_gid(&my_dest->gid, gid);
+	sprintf(msg, "%04x:%06x:%06x:%s", my_dest->lid, my_dest->qpn,
+							my_dest->psn, gid);
 	if (write(sockfd, msg, sizeof msg) != sizeof msg) {
 		fprintf(stderr, "Couldn't send local address\n");
 		goto out;
@@ -175,7 +187,9 @@ static struct pingpong_dest *pp_client_exch_dest(const char *servername, int por
 	if (!rem_dest)
 		goto out;

-	sscanf(msg, "%x:%x:%x", &rem_dest->lid, &rem_dest->qpn, &rem_dest->psn);
+	sscanf(msg, "%x:%x:%x:%s", &rem_dest->lid, &rem_dest->qpn,
+						&rem_dest->psn, gid);
+	wire_gid_to_gid(gid, &rem_dest->gid);

 out:
 	close(sockfd);
@@ -185,7 +199,8 @@ out:
 static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 						 int ib_port, enum ibv_mtu mtu,
 						 int port, int sl,
-						 const struct pingpong_dest *my_dest)
+						 const struct pingpong_dest *my_dest,
+						 int sgid_idx)
 {
 	struct addrinfo *res, *t;
 	struct addrinfo hints = {
@@ -194,10 +209,11 @@ static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 		.ai_socktype = SOCK_STREAM
 	};
 	char *service;
-	char msg[sizeof "0000:000000:000000"];
+	char msg[sizeof "0000:000000:000000:00000000000000000000000000000000"];
 	int n;
 	int sockfd = -1, connfd;
 	struct pingpong_dest *rem_dest = NULL;
+	char gid[33];

 	if (asprintf(&service, "%d", port) < 0)
 		return NULL;
@@ -251,16 +267,22 @@ static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 	if (!rem_dest)
 		goto out;

-	sscanf(msg, "%x:%x:%x", &rem_dest->lid, &rem_dest->qpn, &rem_dest->psn);
+	sscanf(msg, "%x:%x:%x:%s", &rem_dest->lid, &rem_dest->qpn,
+							&rem_dest->psn, gid);
+	wire_gid_to_gid(gid, &rem_dest->gid);

-	if (pp_connect_ctx(ctx, ib_port, my_dest->psn, mtu, sl, rem_dest)) {
+	if (pp_connect_ctx(ctx, ib_port, my_dest->psn, mtu, sl, rem_dest,
+								sgid_idx)) {
 		fprintf(stderr, "Couldn't connect to remote QP\n");
 		free(rem_dest);
 		rem_dest = NULL;
 		goto out;
 	}

-	sprintf(msg, "%04x:%06x:%06x", my_dest->lid, my_dest->qpn, my_dest->psn);
+
+	gid_to_wire_gid(&my_dest->gid, gid);
+	sprintf(msg, "%04x:%06x:%06x:%s", my_dest->lid, my_dest->qpn,
+							my_dest->psn, gid);
 	if (write(connfd, msg, sizeof msg) != sizeof msg) {
 		fprintf(stderr, "Couldn't send local address\n");
 		free(rem_dest);
@@ -281,7 +303,7 @@ static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev, int size,
 {
 	struct pingpong_context *ctx;

-	ctx = malloc(sizeof *ctx);
+	ctx = calloc(1, sizeof *ctx);
 	if (!ctx)
 		return NULL;

@@ -294,7 +316,8 @@ static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev, int size,
 		return NULL;
 	}

-	memset(ctx->buf, 0, size);
+	/* FIXME memset(ctx->buf, 0, size); */
+	memset(ctx->buf, 0x7b, size);

 	ctx->context = ibv_open_device(ib_dev);
 	if (!ctx->context) {
@@ -469,6 +492,7 @@ static void usage(const char *argv0)
 	printf("  -n, --iters=<iters>    number of exchanges (default 1000)\n");
 	printf("  -l, --sl=<sl>          service level value\n");
 	printf("  -e, --events           sleep on CQ events (default poll)\n");
+	printf("  -g, --gid-idx=<gid index> local port gid index\n");
 }

 int main(int argc, char *argv[])
@@ -492,6 +516,8 @@ int main(int argc, char *argv[])
 	int                      rcnt, scnt;
 	int                      num_cq_events = 0;
 	int                      sl = 0;
+	int			 gidx = -1;
+	char			 gid[33];

 	srand48(getpid() * time(NULL));

@@ -508,10 +534,12 @@ int main(int argc, char *argv[])
 			{ .name = "iters",    .has_arg = 1, .val = 'n' },
 			{ .name = "sl",       .has_arg = 1, .val = 'l' },
 			{ .name = "events",   .has_arg = 0, .val = 'e' },
+			{ .name = "gid-idx",  .has_arg = 1, .val = 'g' },
 			{ 0 }
 		};

-		c = getopt_long(argc, argv, "p:d:i:s:m:r:n:l:e", long_options, NULL);
+		c = getopt_long(argc, argv, "p:d:i:s:m:r:n:l:eg:",
+							long_options, NULL);
 		if (c == -1)
 			break;

@@ -564,6 +592,10 @@ int main(int argc, char *argv[])
 			++use_event;
 			break;

+		case 'g':
+			gidx = strtol(optarg, NULL, 0);
+			break;
+
 		default:
 			usage(argv[0]);
 			return 1;
@@ -619,30 +651,50 @@ int main(int argc, char *argv[])
 			return 1;
 		}

-	my_dest.lid = pp_get_local_lid(ctx->context, ib_port);
-	my_dest.qpn = ctx->qp->qp_num;
-	my_dest.psn = lrand48() & 0xffffff;
-	if (!my_dest.lid) {
+
+	if (pp_get_port_info(ctx->context, ib_port, &ctx->portinfo)) {
+		fprintf(stderr, "Couldn't get port info\n");
+		return 1;
+	}
+
+	my_dest.lid = ctx->portinfo.lid;
+	if (ctx->portinfo.link_layer != IBV_LINK_LAYER_ETHERNET &&
+							!my_dest.lid) {
 		fprintf(stderr, "Couldn't get local LID\n");
 		return 1;
 	}

-	printf("  local address:  LID 0x%04x, QPN 0x%06x, PSN 0x%06x\n",
-	       my_dest.lid, my_dest.qpn, my_dest.psn);
+	if (gidx >= 0) {
+		if (ibv_query_gid(ctx->context, ib_port, gidx, &my_dest.gid)) {
+			fprintf(stderr, "can't read sgid of index %d\n", gidx);
+			return 1;
+		}
+	} else
+		memset(&my_dest.gid, 0, sizeof my_dest.gid);
+
+	my_dest.qpn = ctx->qp->qp_num;
+	my_dest.psn = lrand48() & 0xffffff;
+	inet_ntop(AF_INET6, &my_dest.gid, gid, sizeof gid);
+	printf("  local address:  LID 0x%04x, QPN 0x%06x, PSN 0x%06x, GID %s\n",
+	       my_dest.lid, my_dest.qpn, my_dest.psn, gid);
+

 	if (servername)
 		rem_dest = pp_client_exch_dest(servername, port, &my_dest);
 	else
-		rem_dest = pp_server_exch_dest(ctx, ib_port, mtu, port, sl, &my_dest);
+		rem_dest = pp_server_exch_dest(ctx, ib_port, mtu, port, sl,
+								&my_dest, gidx);

 	if (!rem_dest)
 		return 1;

-	printf("  remote address: LID 0x%04x, QPN 0x%06x, PSN 0x%06x\n",
-	       rem_dest->lid, rem_dest->qpn, rem_dest->psn);
+	inet_ntop(AF_INET6, &rem_dest->gid, gid, sizeof gid);
+	printf("  remote address: LID 0x%04x, QPN 0x%06x, PSN 0x%06x, GID %s\n",
+	       rem_dest->lid, rem_dest->qpn, rem_dest->psn, gid);

 	if (servername)
-		if (pp_connect_ctx(ctx, ib_port, my_dest.psn, mtu, sl, rem_dest))
+		if (pp_connect_ctx(ctx, ib_port, my_dest.psn, mtu, sl, rem_dest,
+					gidx))
 			return 1;

 	ctx->pending = PINGPONG_RECV_WRID;
@@ -694,6 +746,7 @@ int main(int argc, char *argv[])
 					fprintf(stderr, "poll CQ failed %d\n", ne);
 					return 1;
 				}
+
 			} while (!use_event && ne < 1);

 			for (i = 0; i < ne; ++i) {
diff --git a/examples/ud_pingpong.c b/examples/ud_pingpong.c
index 6f10212..71b152d 100644
--- a/examples/ud_pingpong.c
+++ b/examples/ud_pingpong.c
@@ -68,16 +68,18 @@ struct pingpong_context {
 	int			 size;
 	int			 rx_depth;
 	int			 pending;
+	struct ibv_port_attr     portinfo;
 };

 struct pingpong_dest {
 	int lid;
 	int qpn;
 	int psn;
+	union ibv_gid gid;
 };

 static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn,
-			  int sl, struct pingpong_dest *dest)
+			  int sl, struct pingpong_dest *dest, int sgid_idx)
 {
 	struct ibv_ah_attr ah_attr = {
 		.is_global     = 0,
@@ -105,6 +107,13 @@ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn,
 		return 1;
 	}

+	if (dest->gid.global.interface_id) {
+		ah_attr.is_global = 1;
+		ah_attr.grh.hop_limit = 1;
+		ah_attr.grh.dgid = dest->gid;
+		ah_attr.grh.sgid_index = sgid_idx;
+	}
+
 	ctx->ah = ibv_create_ah(ctx->pd, &ah_attr);
 	if (!ctx->ah) {
 		fprintf(stderr, "Failed to create AH\n");
@@ -123,10 +132,11 @@ static struct pingpong_dest *pp_client_exch_dest(const char *servername, int por
 		.ai_socktype = SOCK_STREAM
 	};
 	char *service;
-	char msg[sizeof "0000:000000:000000"];
+	char msg[sizeof "0000:000000:000000:00000000000000000000000000000000"];
 	int n;
 	int sockfd = -1;
 	struct pingpong_dest *rem_dest = NULL;
+	char gid[33];

 	if (asprintf(&service, "%d", port) < 0)
 		return NULL;
@@ -157,7 +167,9 @@ static struct pingpong_dest *pp_client_exch_dest(const char *servername, int por
 		return NULL;
 	}

-	sprintf(msg, "%04x:%06x:%06x", my_dest->lid, my_dest->qpn, my_dest->psn);
+	gid_to_wire_gid(&my_dest->gid, gid);
+	sprintf(msg, "%04x:%06x:%06x:%s", my_dest->lid, my_dest->qpn,
+							my_dest->psn, gid);
 	if (write(sockfd, msg, sizeof msg) != sizeof msg) {
 		fprintf(stderr, "Couldn't send local address\n");
 		goto out;
@@ -175,7 +187,9 @@ static struct pingpong_dest *pp_client_exch_dest(const char *servername, int por
 	if (!rem_dest)
 		goto out;

-	sscanf(msg, "%x:%x:%x", &rem_dest->lid, &rem_dest->qpn, &rem_dest->psn);
+	sscanf(msg, "%x:%x:%x:%s", &rem_dest->lid, &rem_dest->qpn,
+							&rem_dest->psn, gid);
+	wire_gid_to_gid(gid, &rem_dest->gid);

 out:
 	close(sockfd);
@@ -184,7 +198,8 @@ out:

 static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 						 int ib_port, int port, int sl,
-						 const struct pingpong_dest *my_dest)
+						 const struct pingpong_dest *my_dest,
+						 int sgid_idx)
 {
 	struct addrinfo *res, *t;
 	struct addrinfo hints = {
@@ -193,10 +208,11 @@ static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 		.ai_socktype = SOCK_STREAM
 	};
 	char *service;
-	char msg[sizeof "0000:000000:000000"];
+	char msg[sizeof "0000:000000:000000:00000000000000000000000000000000"];
 	int n;
 	int sockfd = -1, connfd;
 	struct pingpong_dest *rem_dest = NULL;
+	char gid[33];

 	if (asprintf(&service, "%d", port) < 0)
 		return NULL;
@@ -250,16 +266,21 @@ static struct pingpong_dest *pp_server_exch_dest(struct pingpong_context *ctx,
 	if (!rem_dest)
 		goto out;

-	sscanf(msg, "%x:%x:%x", &rem_dest->lid, &rem_dest->qpn, &rem_dest->psn);
+	sscanf(msg, "%x:%x:%x:%s", &rem_dest->lid, &rem_dest->qpn,
+							&rem_dest->psn, gid);
+	wire_gid_to_gid(gid, &rem_dest->gid);

-	if (pp_connect_ctx(ctx, ib_port, my_dest->psn, sl, rem_dest)) {
+	if (pp_connect_ctx(ctx, ib_port, my_dest->psn, sl, rem_dest,
+								sgid_idx)) {
 		fprintf(stderr, "Couldn't connect to remote QP\n");
 		free(rem_dest);
 		rem_dest = NULL;
 		goto out;
 	}

-	sprintf(msg, "%04x:%06x:%06x", my_dest->lid, my_dest->qpn, my_dest->psn);
+	gid_to_wire_gid(&my_dest->gid, gid);
+	sprintf(msg, "%04x:%06x:%06x:%s", my_dest->lid, my_dest->qpn,
+							my_dest->psn, gid);
 	if (write(connfd, msg, sizeof msg) != sizeof msg) {
 		fprintf(stderr, "Couldn't send local address\n");
 		free(rem_dest);
@@ -293,7 +314,8 @@ static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev, int size,
 		return NULL;
 	}

-	memset(ctx->buf, 0, size + 40);
+	/* FIXME memset(ctx->buf, 0, size + 40); */
+	memset(ctx->buf, 0x7b, size + 40);

 	ctx->context = ibv_open_device(ib_dev);
 	if (!ctx->context) {
@@ -478,6 +500,7 @@ static void usage(const char *argv0)
 	printf("  -r, --rx-depth=<dep>   number of receives to post at a time (default 500)\n");
 	printf("  -n, --iters=<iters>    number of exchanges (default 1000)\n");
 	printf("  -e, --events           sleep on CQ events (default poll)\n");
+	printf("  -g, --gid-idx=<gid index> local port gid index\n");
 }

 int main(int argc, char *argv[])
@@ -500,6 +523,8 @@ int main(int argc, char *argv[])
 	int                      rcnt, scnt;
 	int                      num_cq_events = 0;
 	int                      sl = 0;
+	int			 gidx = -1;
+	char			 gid[33];

 	srand48(getpid() * time(NULL));

@@ -515,10 +540,12 @@ int main(int argc, char *argv[])
 			{ .name = "iters",    .has_arg = 1, .val = 'n' },
 			{ .name = "sl",       .has_arg = 1, .val = 'l' },
 			{ .name = "events",   .has_arg = 0, .val = 'e' },
+			{ .name = "gid-idx",  .has_arg = 1, .val = 'g' },
 			{ 0 }
 		};

-		c = getopt_long(argc, argv, "p:d:i:s:r:n:l:e", long_options, NULL);
+		c = getopt_long(argc, argv, "p:d:i:s:r:n:l:eg:",
+							long_options, NULL);
 		if (c == -1)
 			break;

@@ -563,6 +590,10 @@ int main(int argc, char *argv[])
 			++use_event;
 			break;

+		case 'g':
+			gidx = strtol(optarg, NULL, 0);
+			break;
+
 		default:
 			usage(argv[0]);
 			return 1;
@@ -618,30 +649,44 @@ int main(int argc, char *argv[])
 			return 1;
 		}

-	my_dest.lid = pp_get_local_lid(ctx->context, ib_port);
-	my_dest.qpn = ctx->qp->qp_num;
-	my_dest.psn = lrand48() & 0xffffff;
-	if (!my_dest.lid) {
-		fprintf(stderr, "Couldn't get local LID\n");
+	if (pp_get_port_info(ctx->context, ib_port, &ctx->portinfo)) {
+		fprintf(stderr, "Couldn't get port info\n");
 		return 1;
 	}
+	my_dest.lid = ctx->portinfo.lid;
+
+	my_dest.qpn = ctx->qp->qp_num;
+	my_dest.psn = lrand48() & 0xffffff;
+
+	if (gidx >= 0) {
+		if (ibv_query_gid(ctx->context, ib_port, gidx, &my_dest.gid)) {
+			fprintf(stderr, "Could not get local gid for gid index "
+								"%d\n", gidx);
+			return 1;
+		}
+	} else
+		memset(&my_dest.gid, 0, sizeof my_dest.gid);

-	printf("  local address:  LID 0x%04x, QPN 0x%06x, PSN 0x%06x\n",
-	       my_dest.lid, my_dest.qpn, my_dest.psn);
+	inet_ntop(AF_INET6, &my_dest.gid, gid, sizeof gid);
+	printf("  local address:  LID 0x%04x, QPN 0x%06x, PSN 0x%06x: GID %s\n",
+	       my_dest.lid, my_dest.qpn, my_dest.psn, gid);

 	if (servername)
 		rem_dest = pp_client_exch_dest(servername, port, &my_dest);
 	else
-		rem_dest = pp_server_exch_dest(ctx, ib_port, port, sl, &my_dest);
+		rem_dest = pp_server_exch_dest(ctx, ib_port, port, sl,
+							&my_dest, gidx);

 	if (!rem_dest)
 		return 1;

-	printf("  remote address: LID 0x%04x, QPN 0x%06x, PSN 0x%06x\n",
-	       rem_dest->lid, rem_dest->qpn, rem_dest->psn);
+	inet_ntop(AF_INET6, &rem_dest->gid, gid, sizeof gid);
+	printf("  remote address: LID 0x%04x, QPN 0x%06x, PSN 0x%06x, GID %s\n",
+	       rem_dest->lid, rem_dest->qpn, rem_dest->psn, gid);

 	if (servername)
-		if (pp_connect_ctx(ctx, ib_port, my_dest.psn, sl, rem_dest))
+		if (pp_connect_ctx(ctx, ib_port, my_dest.psn, sl, rem_dest,
+									gidx))
 			return 1;

 	ctx->pending = PINGPONG_RECV_WRID;
-- 
1.5.5


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 5/7] libmlx4: add IBoE support

[Or Gerlitz]

Modify libmlx4 to support IBoE, where the only user space piece
to handle is the creation of UD address handles - the L2 Ethernet
attributes have to be resolved from the DGID.
Derived from work by Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>

Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 src/mlx4.h  |    1 +
 src/qp.c    |    1 +
 src/verbs.c |   49 +++++++++++++++++++++++++++++++++++++++++++++++--
 src/wqe.h   |    3 ++-
 4 files changed, 51 insertions(+), 3 deletions(-)

diff --git a/src/mlx4.h b/src/mlx4.h
index 4445998..b277b06 100644
--- a/src/mlx4.h
+++ b/src/mlx4.h
@@ -241,6 +241,7 @@ struct mlx4_av {
 struct mlx4_ah {
 	struct ibv_ah			ibv_ah;
 	struct mlx4_av			av;
+	uint8_t				mac[6];
 };

 static inline unsigned long align(unsigned long val, unsigned long align)
diff --git a/src/qp.c b/src/qp.c
index ec138cd..4d79e38 100644
--- a/src/qp.c
+++ b/src/qp.c
@@ -144,6 +144,7 @@ static void set_datagram_seg(struct mlx4_wqe_datagram_seg *dseg,
 	memcpy(dseg->av, &to_mah(wr->wr.ud.ah)->av, sizeof (struct mlx4_av));
 	dseg->dqpn = htonl(wr->wr.ud.remote_qpn);
 	dseg->qkey = htonl(wr->wr.ud.remote_qkey);
+	memcpy(dseg->mac, to_mah(wr->wr.ud.ah)->mac, 6);
 }

 static void __set_data_seg(struct mlx4_wqe_data_seg *dseg, struct ibv_sge *sg)
diff --git a/src/verbs.c b/src/verbs.c
index 1ac1362..6620ac2 100644
--- a/src/verbs.c
+++ b/src/verbs.c
@@ -614,9 +614,45 @@ int mlx4_destroy_qp(struct ibv_qp *ibqp)
 	return 0;
 }

+static int link_local_gid(const union ibv_gid *gid)
+{
+	uint32_t hi = *(uint32_t *)(gid->raw);
+	uint32_t lo = *(uint32_t *)(gid->raw + 4);
+	if (hi == htonl(0xfe800000) && lo == 0)
+		return 1;
+
+	return 0;
+}
+
+static uint16_t get_vlan_id(union ibv_gid *gid)
+{
+	uint16_t vid;
+	vid = gid->raw[11] << 8 | gid->raw[12];
+	return vid < 0x1000 ? vid : 0xffff;
+}
+
+
+static int mlx4_resolve_grh_to_l2(struct mlx4_ah *ah, struct ibv_ah_attr *attr)
+{
+	if (get_vlan_id(&attr->grh.dgid) != 0xffff)
+		return 1;
+
+	if (link_local_gid(&attr->grh.dgid)) {
+		memcpy(ah->mac, &attr->grh.dgid.raw[8], 3);
+		memcpy(ah->mac + 3, &attr->grh.dgid.raw[13], 3);
+		ah->mac[0] ^= 2;
+		return 0;
+	} else
+		return 1;
+}
+
 struct ibv_ah *mlx4_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr)
 {
 	struct mlx4_ah *ah;
+	struct ibv_port_attr port_attr;
+
+	if (ibv_query_port(pd->context, attr->port_num, &port_attr))
+		return NULL;

 	ah = malloc(sizeof *ah);
 	if (!ah)
@@ -625,8 +661,11 @@ struct ibv_ah *mlx4_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr)
 	memset(&ah->av, 0, sizeof ah->av);

 	ah->av.port_pd   = htonl(to_mpd(pd)->pdn | (attr->port_num << 24));
-	ah->av.g_slid    = attr->src_path_bits;
-	ah->av.dlid      = htons(attr->dlid);
+
+	if (port_attr.link_layer != IBV_LINK_LAYER_ETHERNET) {
+		ah->av.g_slid = attr->src_path_bits;
+		ah->av.dlid   = htons(attr->dlid);
+	}
 	if (attr->static_rate) {
 		ah->av.stat_rate = attr->static_rate + MLX4_STAT_RATE_OFFSET;
 		/* XXX check rate cap? */
@@ -642,6 +681,12 @@ struct ibv_ah *mlx4_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr)
 		memcpy(ah->av.dgid, attr->grh.dgid.raw, 16);
 	}

+	if (port_attr.link_layer == IBV_LINK_LAYER_ETHERNET)
+		if (mlx4_resolve_grh_to_l2(ah, attr)) {
+			free(ah);
+			return NULL;
+		}
+
 	return &ah->ibv_ah;
 }

diff --git a/src/wqe.h b/src/wqe.h
index 6f7f309..043f0da 100644
--- a/src/wqe.h
+++ b/src/wqe.h
@@ -78,7 +78,8 @@ struct mlx4_wqe_datagram_seg {
 	uint32_t		av[8];
 	uint32_t		dqpn;
 	uint32_t		qkey;
-	uint32_t		reserved[2];
+	uint16_t		reserved;
+	uint8_t			mac[6];
 };

 struct mlx4_wqe_data_seg {
-- 
1.5.5


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 6/7] libmlx4: add IBoE UD/VLANs support

[Or Gerlitz]

Add VLANs support for the UD address handle creation flow,
where the vlan id is taken from the destination GID and the
and vlan priority from the IB SL specified by the application.

Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 src/mlx4.h  |    1 +
 src/qp.c    |    1 +
 src/verbs.c |    9 +++++++--
 src/wqe.h   |    2 +-
 4 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/src/mlx4.h b/src/mlx4.h
index b277b06..0ad838d 100644
--- a/src/mlx4.h
+++ b/src/mlx4.h
@@ -241,6 +241,7 @@ struct mlx4_av {
 struct mlx4_ah {
 	struct ibv_ah			ibv_ah;
 	struct mlx4_av			av;
+	uint16_t			vlan;
 	uint8_t				mac[6];
 };

diff --git a/src/qp.c b/src/qp.c
index 4d79e38..40a6689 100644
--- a/src/qp.c
+++ b/src/qp.c
@@ -144,6 +144,7 @@ static void set_datagram_seg(struct mlx4_wqe_datagram_seg *dseg,
 	memcpy(dseg->av, &to_mah(wr->wr.ud.ah)->av, sizeof (struct mlx4_av));
 	dseg->dqpn = htonl(wr->wr.ud.remote_qpn);
 	dseg->qkey = htonl(wr->wr.ud.remote_qkey);
+	dseg->vlan = htons(to_mah(wr->wr.ud.ah)->vlan);
 	memcpy(dseg->mac, to_mah(wr->wr.ud.ah)->mac, 6);
 }

diff --git a/src/verbs.c b/src/verbs.c
index 6620ac2..8aa9860 100644
--- a/src/verbs.c
+++ b/src/verbs.c
@@ -634,13 +634,18 @@ static uint16_t get_vlan_id(union ibv_gid *gid)

 static int mlx4_resolve_grh_to_l2(struct mlx4_ah *ah, struct ibv_ah_attr *attr)
 {
-	if (get_vlan_id(&attr->grh.dgid) != 0xffff)
-		return 1;
+	uint16_t vid;

 	if (link_local_gid(&attr->grh.dgid)) {
 		memcpy(ah->mac, &attr->grh.dgid.raw[8], 3);
 		memcpy(ah->mac + 3, &attr->grh.dgid.raw[13], 3);
 		ah->mac[0] ^= 2;
+
+		vid = get_vlan_id(&attr->grh.dgid);
+		if (vid != 0xffff) {
+			ah->av.port_pd |= htonl(1 << 29);
+			ah->vlan = vid | ((attr->sl & 7) << 13);
+		}
 		return 0;
 	} else
 		return 1;
diff --git a/src/wqe.h b/src/wqe.h
index 043f0da..bbd22ba 100644
--- a/src/wqe.h
+++ b/src/wqe.h
@@ -78,7 +78,7 @@ struct mlx4_wqe_datagram_seg {
 	uint32_t		av[8];
 	uint32_t		dqpn;
 	uint32_t		qkey;
-	uint16_t		reserved;
+	uint16_t		vlan;
 	uint8_t			mac[6];
 };

-- 
1.5.5


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 7/7] libmlx4: align the list of ConnectX devices supported with the kernel

[Or Gerlitz]

align the list of ConnectX devices supported by the library to be
the same as the mlx4 driver from the upstream kernel

Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---

Roland, these two simple awk/cut commands can be used to
actually validate the claim made by the change log...

$ grep MELLANOX libmlx4.git/src/mlx4.c | grep HCA | awk '{ print $2 }' | cut -d ")" -f 1 > lib
$ grep MELLANOX linux-2.6.git/drivers/net/mlx4/main.c | awk '{ print $3 }' | cut -d ")" -f 1 > ker
$ diff lib ker

The patch previously posted @ http://www.spinics.net/lists/linux-rdma/msg07237.html
contains also the VPI/DDR device which isn't present in the kernel (device id 0x6778).
Once the table cleanup brought by this patch takes place, we can add further
patches to both the library and the kernel e.g with that DDR device, etc.

 src/mlx4.c |   23 +++++++++++++++++++++++
 1 files changed, 23 insertions(+), 0 deletions(-)

diff --git a/src/mlx4.c b/src/mlx4.c
index 1295c53..ab18b4a 100644
--- a/src/mlx4.c
+++ b/src/mlx4.c
@@ -66,8 +66,31 @@ struct {
 	HCA(MELLANOX, 0x6354),	/* MT25408 "Hermon" QDR */
 	HCA(MELLANOX, 0x6732),	/* MT25408 "Hermon" DDR PCIe gen2 */
 	HCA(MELLANOX, 0x673c),	/* MT25408 "Hermon" QDR PCIe gen2 */
+	HCA(MELLANOX, 0x6368),	/* MT25408 "Hermon" EN 10GigE */
+	HCA(MELLANOX, 0x6750),	/* MT25408 "Hermon" EN 10GigE PCIe gen2 */
+	HCA(MELLANOX, 0x6372),	/* MT25458 ConnectX EN 10GBASE-T 10GigE */
+	HCA(MELLANOX, 0x675a),	/* MT25458 ConnectX EN 10GBASE-T+Gen2 10GigE */
+	HCA(MELLANOX, 0x6764),	/* MT26468 ConnectX EN 10GigE PCIe gen2*/
+	HCA(MELLANOX, 0x6746),	/* MT26438 ConnectX EN 40GigE PCIe gen2 5GT/s */
+	HCA(MELLANOX, 0x676e),	/* MT26478 ConnectX2 40GigE PCIe gen2 */
+	HCA(MELLANOX, 0x1002),	/* MT25400 Family [ConnectX-2 Virtual Function] */
+	HCA(MELLANOX, 0x1003),	/* MT27500 Family [ConnectX-3] */
+	HCA(MELLANOX, 0x1004),	/* MT27500 Family [ConnectX-3 Virtual Function] */
+	HCA(MELLANOX, 0x1005),	/* MT27510 Family */
+	HCA(MELLANOX, 0x1006),	/* MT27511 Family */
+	HCA(MELLANOX, 0x1007),	/* MT27520 Family */
+	HCA(MELLANOX, 0x1008),	/* MT27521 Family */
+	HCA(MELLANOX, 0x1009),	/* MT27530 Family */
+	HCA(MELLANOX, 0x100a),	/* MT27531 Family */
+	HCA(MELLANOX, 0x100b),	/* MT27540 Family */
+	HCA(MELLANOX, 0x100c),	/* MT27541 Family */
+	HCA(MELLANOX, 0x100d),	/* MT27550 Family */
+	HCA(MELLANOX, 0x100e),	/* MT27551 Family */
+	HCA(MELLANOX, 0x100f),	/* MT27560 Family */
+	HCA(MELLANOX, 0x1010),	/* MT27561 Family */
 };

+
 static struct ibv_context_ops mlx4_ctx_ops = {
 	.query_device  = mlx4_query_device,
 	.query_port    = mlx4_query_port,
-- 
1.5.5


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 5/7] libmlx4: add IBoE support

[Jason Gunthorpe]

On Tue, Jul 19, 2011 at 12:32:52PM +0300, Or Gerlitz wrote:
> Modify libmlx4 to support IBoE, where the only user space piece
> to handle is the creation of UD address handles - the L2 Ethernet
> attributes have to be resolved from the DGID.

It seems a shame to add a kernel syscall to every ah creation just to
learn the link type.

Other than rejecting some GIDs it doesn't seem like there is any
operational need to learn the link type..

Also, did anyone document the GID format for IBoE? IIRC we invented
this for Linux since IBA refused to standardize anything.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/7] libibverbs: Update examples for IBoE

[Jason Gunthorpe]

On Tue, Jul 19, 2011 at 12:31:32PM +0300, Or Gerlitz wrote:
> Since IBoE requires usage of GRH, update ibv_*_pinpong examples to accept
> GIDs. GIDs are given as an index to the local port's table and are exchanged
> between the client and the server through the socket connection.

Since these examples are meant as programming examples, using the GID
index as an command line parameter is not an example of good UI
design. Other tools (ie diags, ibtool, etc) accept the GID in
canonical IPv6 format and match that to the GID table.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/7] libibverbs: add GID change event

[Jason Gunthorpe]

On Tue, Jul 19, 2011 at 12:30:08PM +0300, Or Gerlitz wrote:
> Add IB GID change event which is generated by the kernel
> IBoE stack when the HW driver updates the GID table.

You also need to update ibv_event_type_str

Has a kernel patch that reserves/uses the corresponding ID been
accepted yet?

BTW, this has value for straight IB, hopefully it triggers on IB
when the GUID table is altered and when the subnet prefix is changed...

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Jason Gunthorpe]

On Tue, Jul 19, 2011 at 12:27:17PM +0300, Or Gerlitz wrote:

> The addition of the new field does not change the size of struct
> ibv_port_attr due to alignment of the preceding fields. As such
> binary compatibility between the library to applications is kept,

What zeros the field in the current library?

It looks like new apps that want to access the link_layer field will
have to explicitly zero port_attr before calling
ibv_cmd_query_port. Should be noted in the documentation - and maybe
also explicitly zero the remaining pad for future.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 5/7] libmlx4: add IBoE support

[Or Gerlitz]

Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
>> Modify libmlx4 to support IBoE, where the only user space piece
>> to handle is the creation of UD address handles - the L2 Ethernet
>> attributes have to be resolved from the DGID.

> Other than rejecting some GIDs it doesn't seem like there is any
> operational need to learn the link type..

Jason, as the change log explains, under Ethernet link type the code
(see mlx4_resolve_grh_to_l2) goes and extracts the Ethernet L2 info
(mac and vlan, look on the last patch) from the GID.

> It seems a shame to add a kernel syscall to every ah creation just to learn the link type

AH creation is app's slow path, e.g the equivalent of session
creation, so I wouldn't bother too much on perf, on the being elegant
side, I'll check if/how I can plant/cache the link type for the ah
creation code to use.

> Also, did anyone document the GID format for IBoE? IIRC we invented
> this for Linux since IBA refused to standardize anything.

Not really, maybe we can/should add a document to the kernel IB
documentation folder with some details - as this data is more for deep
divers -  the thing is that applications that use the rdma-cm don't
deal directly with the GIDs, they just get it from the kernel and hand
it over to libibverbs (e.g UD apps use address / route resolution and
then call ah create, RC apps even don't have to call libibiverbs, as
the qp modifications go beyond the cover of librdmacm calling
libibverbs).

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Or Gerlitz]

Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:

> What zeros the field in the current library?

the kernel uverbs code either zeros it (old kernels) or sets it (new kernels)


Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/7] libibverbs: add GID change event

[Or Gerlitz]

Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:

> You also need to update ibv_event_type_str

sure, will do that.

> Has a kernel patch that reserves/uses the corresponding ID been accepted yet?

yes, its in Roland's for-next branch

> BTW, this has value for straight IB, hopefully it triggers on IB
> when the GUID table is altered and when the subnet prefix is changed...

The kernel is triggering this event when the GID table is changed from
the IBoE code path, the library is just receiving the event and
delivering it over to the application.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/7] libibverbs: Update examples for IBoE

[Or Gerlitz]

Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
>> Since IBoE requires usage of GRH, update ibv_*_pinpong examples to accept
>> GIDs. GIDs are given as an index to the local port's table and are exchanged
>> between the client and the server through the socket connection.

> Since these examples are meant as programming examples, using the GID
> index as an command line parameter is not an example of good UI
> design. Other tools (ie diags, ibtool, etc) accept the GID in
> canonical IPv6 format and match that to the GID table.

To my taste these examples should be used as programming examples for
their verbs
data path part, as for connection establishment, I would prefer app
writers use the rdma-cm and I don't think the control plane of these
examples (the IB L2/L3/L4 info exchange) should be used in real life
apps. In the same manner that under IB the user doesn't specify the
LID from the command line but rather the device and port, I don't
think they need to specify the GID but rather the device and port. The
index isn't always zero, since this is what let people test VLANs
(under IBoE for each VLAN a new entry is added to the GID table as the
GID includes both mac and vlan).

Or.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Jason Gunthorpe]

On Tue, Jul 19, 2011 at 10:36:26PM +0300, Or Gerlitz wrote:
> > What zeros the field in the current library?
> 
> the kernel uverbs code either zeros it (old kernels) or sets it (new kernels)

ibv_cmd_query_port does a field by field copy of the port_attr, so
what the kernel does is not relevant to this case, and to me it looks
like the new value you added will be left uninitialized on old ibverbs.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 5/7] libmlx4: add IBoE support

[Or Gerlitz]

Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> Or Gerlitz wrote:

>> Jason, as the change log explains, under Ethernet link type the code
>> (see mlx4_resolve_grh_to_l2) goes and extracts the Ethernet L2 info
>> (mac and vlan, look on the last patch) from the GID.
>
> Looking at the control flow of the actual code, there is no reason it
> needs to know the link type, except to generate that error code. Why
> can't it set the slid, dlid and MAC fields in mlx4_ah simultaneously?

If you look on patch 6/7 @ mlx4_resolve_grh_to_l2, you would see this section

+               if (vid != 0xffff) {
+                       ah->av.port_pd |= htonl(1 << 29);
+                       ah->vlan = vid | ((attr->sl & 7) << 13);
+               }

this bit in the PD tells the HW  to add VLAN header to the packet and
sets the user priority bits for that vlan from the SL, we want to do
that only for Ethernet link type.

>> AH creation is app's slow path, e.g the equivalent of session
>> creation, so I wouldn't bother too much on perf, on the being elegant
>> side, I'll check if/how I can plant/cache the link type for the ah
>> creation code to use.

> Still, the MPI people regularly seem to have job startup problems, no
> idea how many AH's they end up creating at startup though..


Under IB job startup problems relates to SA scalability, generally under any
transport @ large scale there could be more issues, I don't think AH
creation time
is among them.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Or Gerlitz]

Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> Or Gerlitz wrote:

>>> What zeros the field in the current library?

+       port_attr->link_layer = IBV_LINK_LAYER_UNSPECIFIED;
+       port_attr->pad = 0;

Or
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Jason Gunthorpe]

On Tue, Jul 19, 2011 at 11:22:51PM +0300, Or Gerlitz wrote:
> Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > Or Gerlitz wrote:
> 
> >>> What zeros the field in the current library?
> 
> +       port_attr->link_layer = IBV_LINK_LAYER_UNSPECIFIED;
> +       port_attr->pad = 0;

Where are you getting that hunk from?

libibverbs $ git grep link_layer libibverbs-1.1.5 --
libibverbs $

More clearly: what initializes the memory you've redefined to be
link_layer in libibverbs 1.1.5 ? I see nothing.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Hefty, Sean]

> More clearly: what initializes the memory you've redefined to be
> link_layer in libibverbs 1.1.5 ? I see nothing.

Why does the value in an older library cause an issue?

Or, how will this library version interact with an older kernel?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Jason Gunthorpe]

On Tue, Jul 19, 2011 at 09:20:25PM +0000, Hefty, Sean wrote:
> > More clearly: what initializes the memory you've redefined to be
> > link_layer in libibverbs 1.1.5 ? I see nothing.
> 
> Why does the value in an older library cause an issue?

ABI compatability means if dynamic linking succeeds then the result
works, so the behavior of a new app, using new functionality when it
links to the old library must work.

The commit comment states the goal is to have new apps see
IBV_LINK_LAYER_UNSPECIFIED for all possible combinations of old stuff
underneath.

> Or, how will this library version interact with an older kernel?

The kernel interaction is not what I'm talking about - AFAIK, the
kernel interaction is OK. Old kernels zero the entire ib_port_attr
structure before copy_to_user so the reserved bytes are guaranteed to
be 0.

The problem is with the userspace ABI out of libibverbs. Current
libibverbs does not zero its ib_port_attr structure before returning.

Build this with Or's patch applied, then run it against libibverbs 1.1.5

 struct ibv_port_attr attr;
 attr.link_layer = 100;
 ibv_query_port(..,&attr);
 assert(attr.link_layer == IBV_LINK_LAYER_UNSPECIFIED); // Fails!

That is an ABI breaking issue that is not dealt with by Or's patch.

Couple choices:
 1) Require and document callers using ibv_query_port to zero attr
    before hand when using link_layer
 2) Crank the symbol version on ibv_query_port so dynamic linking
    fails
 3) Inline the memset in ibverbs.h:

    static inline int fixed_ibv_query_port(...)
    {
       attr->link_layer = 0;
       return ibv_query_port(...);
    }
    #define ibv_query_port(...) fixed_ibv_query_port(...)

#2 might be the best?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Or Gerlitz]

Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:

> The commit comment states the goal is to have new apps see
> IBV_LINK_LAYER_UNSPECIFIED for all possible combinations of old stuff underneath.
> [...]  The problem is with the userspace ABI out of libibverbs. Current
> libibverbs does not zero its ib_port_attr structure before returning.

> Build this with Or's patch applied, then run it against libibverbs 1.1.5
>  struct ibv_port_attr attr;
>  attr.link_layer = 100;
>  ibv_query_port(..,&attr);
>  assert(attr.link_layer == IBV_LINK_LAYER_UNSPECIFIED); // Fails!

impossible, see below why

> That is an ABI breaking issue that is not dealt with by Or's patch - couple choices:
>  1) Require and document callers using ibv_query_port to zero attr
>    before hand when using link_layer
>  2) Crank the symbol version on ibv_query_port so dynamic linking fails
>  3) Inline the memset in ibverbs.h:


YES, #3 is what the patch does, and it follows your suggestion!!

+static inline int ___ibv_query_port(struct ibv_context *context,
+                                   uint8_t port_num,
+                                   struct ibv_port_attr *port_attr)
+{
+       port_attr->link_layer = IBV_LINK_LAYER_UNSPECIFIED;
+       port_attr->pad = 0;
+
+       return context->ops.query_port(context, port_num, port_attr);
+}
+


+#define ibv_query_port(context, port_num, port_attr) \
+       ___ibv_query_port(context, port_num, port_attr)
+

So we are okay now, correct?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Jason Gunthorpe]

On Wed, Jul 20, 2011 at 09:08:26PM +0300, Or Gerlitz wrote:
> Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:

> +#define ibv_query_port(context, port_num, port_attr) \
> +       ___ibv_query_port(context, port_num, port_attr)
> +
> 
> So we are okay now, correct?

Looks like! Sorry for the confusion

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Or Gerlitz]

Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> Or Gerlitz wrote:
>> Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:

>> +#define ibv_query_port(context, port_num, port_attr) \
>> +       ___ibv_query_port(context, port_num, port_attr)
>> +
>> So we are okay now, correct?

> Looks like! Sorry for the confusion

Jason, thanks for your detailed review && feedback, just to recap, the patches
are okay except the missing entry for the new GID change event in
ibv_event_type_str which I will add, correct?

Sean, any comments/questions?



Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Hefty, Sean]

> Sean, any comments/questions?

I was most concerned that this worked with an older kernel, but Jason answered that.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Jason Gunthorpe]

On Wed, Jul 20, 2011 at 09:20:41PM +0300, Or Gerlitz wrote:
> Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:

> > Looks like! Sorry for the confusion
> 
> Jason, thanks for your detailed review && feedback, just to recap,
> the patches are okay except the missing entry for the new GID change
> event in ibv_event_type_str which I will add, correct?

Yes, that was the only important thing I saw.

I assume Roland was happy with the #define trick for compatibility
last time it came up?

Looking at the diff, maybe re-shuffle the order of the #define trick
so it is easier to follow, and call ibv_query_port rather than short
cut through the context for better future compatibility.

diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index 0f1cb2e..40e8802 100644
@@ -766,6 +774,18 @@ int ibv_query_device(struct ibv_context *context,
  */
 int ibv_query_port(struct ibv_context *context, uint8_t port_num,
 		   struct ibv_port_attr *port_attr);
+static inline int ___ibv_query_port(struct ibv_context *context,
+				    uint8_t port_num,
+				    struct ibv_port_attr *port_attr)
+{
+	/* For compatability linking with old libibverbs */
+	port_attr->link_layer = IBV_LINK_LAYER_UNSPECIFIED;
+	port_attr->pad = 0;
+
+	return ibv_query_port(context,port_num,port_attr);
+}
+#define ibv_query_port(context, port_num, port_attr) \
+	___ibv_query_port(context, port_num, port_attr)
 
 /**
  * ibv_query_gid - Get a GID table entry
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/7] libibverbs: Add link layer field port attribute

[Or Gerlitz]

Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:

> Yes, that was the only important thing I saw.
> I assume Roland was happy with the #define trick for compatibility
> last time it came up?

yes, that's correct.

> Looking at the diff, maybe re-shuffle the order of the #define trick
> so it is easier to follow, and call ibv_query_port rather than short
> cut through the context for better future compatibility.

will do, thanks

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 1/7] libibverbs: Add link layer field port attribute

[Or Gerlitz]

The new field has one of three values - IBV_LINK_LAYER_UNSPECIFIED,
IBV_LINK_LAYER_INFINIBAND, IBV_LINK_LAYER_ETHERNET. It can be used
by applications to know the link layer used by the port, which
can be either Infiniband or Ethernet.

The addition of the new field does not change the size of struct
ibv_port_attr due to alignment of the preceding fields. As such
binary compatibility between the library to applications is kept,
since old apps running over new library do not read this field,
and new apps running over old library will determine the link
layer as unspecified and hence take their IB code path.

The solution was suggested by: Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org>
and Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>

Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
Signed-off-by: Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
---
changes from V0:

- applied feedback from Jason, made the ___ibv_query_port trick
  clearer to header file readers

 include/infiniband/verbs.h |   23 +++++++++++++++++++++++
 man/ibv_query_port.3       |    4 ++++
 2 files changed, 27 insertions(+), 0 deletions(-)

diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index 0f1cb2e..d1a3e47 100644
--- a/include/infiniband/verbs.h
+++ b/include/infiniband/verbs.h
@@ -161,6 +161,12 @@ enum ibv_port_state {
 	IBV_PORT_ACTIVE_DEFER	= 5
 };

+enum {
+	IBV_LINK_LAYER_UNSPECIFIED,
+	IBV_LINK_LAYER_INFINIBAND,
+	IBV_LINK_LAYER_ETHERNET,
+};
+
 struct ibv_port_attr {
 	enum ibv_port_state	state;
 	enum ibv_mtu		max_mtu;
@@ -181,6 +187,8 @@ struct ibv_port_attr {
 	uint8_t			active_width;
 	uint8_t			active_speed;
 	uint8_t			phys_state;
+	uint8_t			link_layer;
+	uint8_t			pad;
 };

 enum ibv_event_type {
@@ -767,6 +775,20 @@ int ibv_query_device(struct ibv_context *context,
 int ibv_query_port(struct ibv_context *context, uint8_t port_num,
 		   struct ibv_port_attr *port_attr);

+static inline int ___ibv_query_port(struct ibv_context *context,
+				    uint8_t port_num,
+				    struct ibv_port_attr *port_attr)
+{
+	/* For compatability linking with old libibverbs */
+	port_attr->link_layer = IBV_LINK_LAYER_UNSPECIFIED;
+	port_attr->pad = 0;
+
+	return ibv_query_port(context, port_num, port_attr);
+}
+
+#define ibv_query_port(context, port_num, port_attr) \
+	___ibv_query_port(context, port_num, port_attr)
+
 /**
  * ibv_query_gid - Get a GID table entry
  */
@@ -1097,4 +1119,5 @@ END_C_DECLS

 #  undef __attribute_const

+
 #endif /* INFINIBAND_VERBS_H */
diff --git a/man/ibv_query_port.3 b/man/ibv_query_port.3
index 882470d..9bedd90 100644
--- a/man/ibv_query_port.3
+++ b/man/ibv_query_port.3
@@ -44,9 +44,13 @@ uint8_t                 init_type_reply;/* Type of initialization performed by S
 uint8_t                 active_width;   /* Currently active link width */
 uint8_t                 active_speed;   /* Currently active link speed */
 uint8_t                 phys_state;     /* Physical port state */
+uint8_t                 link_layer;     /* link layer protocol of the port */
 .in -8
 };
 .sp
+possible values for the link layer field are IBV_LINK_LAYER_INFINIBAND,
+IBV_LINK_LAYER_ETHERNET, or IBV_LINK_LAYER_UNSPECIFIED.
+.sp
 .fi
 .SH "RETURN VALUE"
 .B ibv_query_port()
-- 
1.5.5


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH V1 3/7] libibverbs: add GID change event

[Or Gerlitz]

Add IB GID change event which is generated by the kernel
IBoE stack when the HW driver updates the GID table.

Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
Signed-off-by: Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>

---
changes from V0:

- applied feedback from Jason, added entry to ibv_event_type_str

 examples/asyncwatch.c      |    2 ++
 include/infiniband/verbs.h |    3 ++-
 man/ibv_get_async_event.3  |    2 ++
 src/enum_strs.c            |    3 ++-
 4 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/examples/asyncwatch.c b/examples/asyncwatch.c
index 5510a29..da7ebd4 100644
--- a/examples/asyncwatch.c
+++ b/examples/asyncwatch.c
@@ -57,6 +57,8 @@ static const char *event_name_str(enum ibv_event_type event_type)
 		return "IBV_EVENT_SM_CHANGE";
 	case IBV_EVENT_CLIENT_REREGISTER:
 		return "IBV_EVENT_CLIENT_REREGISTER";
+	case IBV_EVENT_GID_CHANGE:
+		return "IBV_EVENT_GID_CHANGE";

 	case IBV_EVENT_CQ_ERR:
 	case IBV_EVENT_QP_FATAL:
diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index d1a3e47..4892d12 100644
--- a/include/infiniband/verbs.h
+++ b/include/infiniband/verbs.h
@@ -209,7 +209,8 @@ enum ibv_event_type {
 	IBV_EVENT_SRQ_ERR,
 	IBV_EVENT_SRQ_LIMIT_REACHED,
 	IBV_EVENT_QP_LAST_WQE_REACHED,
-	IBV_EVENT_CLIENT_REREGISTER
+	IBV_EVENT_CLIENT_REREGISTER,
+	IBV_EVENT_GID_CHANGE,
 };

 struct ibv_async_event {
diff --git a/man/ibv_get_async_event.3 b/man/ibv_get_async_event.3
index acb6257..a76dc0c 100644
--- a/man/ibv_get_async_event.3
+++ b/man/ibv_get_async_event.3
@@ -81,6 +81,8 @@ following events:
 .B IBV_EVENT_SM_CHANGE \fR SM was changed on a port
 .TP
 .B IBV_EVENT_CLIENT_REREGISTER \fR SM sent a CLIENT_REREGISTER request to a port
+.TP
+.B IBV_EVENT_GID_CHANGE \fR GID table was changed on a port
 .PP
 .I CA events:
 .TP
diff --git a/src/enum_strs.c b/src/enum_strs.c
index c57feaa..54d71a6 100644
--- a/src/enum_strs.c
+++ b/src/enum_strs.c
@@ -85,9 +85,10 @@ const char *ibv_event_type_str(enum ibv_event_type event)
 		[IBV_EVENT_SRQ_LIMIT_REACHED]	= "SRQ limit reached",
 		[IBV_EVENT_QP_LAST_WQE_REACHED]	= "last WQE reached",
 		[IBV_EVENT_CLIENT_REREGISTER]	= "client reregistration",
+		[IBV_EVENT_GID_CHANGE]		= "GID table change"
 	};

-	if (event < IBV_EVENT_CQ_ERR || event > IBV_EVENT_CLIENT_REREGISTER)
+	if (event < IBV_EVENT_CQ_ERR || event > IBV_EVENT_GID_CHANGE)
 		return "unknown";

 	return event_type_str[event];
-- 
1.5.5


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] IBoE user space support

[Or Gerlitz]

Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:

> the enclosed patch series enhances the user space IB stack to support IBoE.

> [PATCH 1/7] libibverbs: Add link layer field port attribute
> [PATCH 2/7] libibverbs: change kernel API to accept link layer
> [PATCH 3/7] libibverbs: add GID change event
> [PATCH 4/7] libibverbs: update examples for IBoE
> [PATCH 5/7] libmlx4: add IBoE support
> [PATCH 6/7] libmlx4: add IBoE UD/VLANs support
> [PATCH 7/7] libmlx4: align the list of ConnectX devices supported with the kernel

Hi Roland,

Following a review by Jason and Sean I've posted V1 for patches 1/7 and 3/7
as replies on the relevant posting of the V0 (this) thread.

Please let me know if you have any further comments/questions on the series

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] IBoE user space support

[Or Gerlitz]

On Wed, Jul 20, 2011 at 10:43 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> the enclosed patch series enhances the user space IB stack to support IBoE.
>> [PATCH 1/7] libibverbs: Add link layer field port attribute
>> [PATCH 2/7] libibverbs: change kernel API to accept link layer
>> [PATCH 3/7] libibverbs: add GID change event
>> [PATCH 4/7] libibverbs: update examples for IBoE
>> [PATCH 5/7] libmlx4: add IBoE support
>> [PATCH 6/7] libmlx4: add IBoE UD/VLANs support
>> [PATCH 7/7] libmlx4: align the list of ConnectX devices supported with the kernel

> Hi Roland, Following a review by Jason and Sean I've posted V1
> for patches 1/7 and 3/7 as replies on the relevant posting of the V0 (this) thread.
> Please let me know if you have any further comments/questions on the series

Hi Roland, just wanted to check if you going to take that series soon,
specifically, soon enough so we have headroom to review/merge code
towards the 3.2 kernel window.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] IBoE user space support

[Roland Dreier]

> Hi Roland, just wanted to check if you going to take that series soon,
> specifically, soon enough so we have headroom to review/merge code
> towards the 3.2 kernel window.

All the IBoE userspace patches should be applied and pushed out.
Let me know if I missed anything.  Of course I can't push anything
more until master.kernel.org is brought back up...

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/7] IBoE user space support

[Or Gerlitz]

> All the IBoE userspace patches should be applied and pushed out.
> Let me know if I missed anything.

Yes, thanks, I see them now, I just didn't notice an explicit ack from
you over the list which made me think there's not there yet, sure,
I'll test with clones of your trees and let you know.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html