[PATCH rdma-core 0/3] Verbs support for source QPN

[PATCH rdma-core 0/3] Verbs support for source QPN

[Yishai Hadas]

This patch set is the supplementary part of the kernel series that was accepted
upstream into 4.14.

It enables user space applications to create a QP with a given source QP
number. The created QP will use the source QPN as its wire QP number.

This functionality is implemented by mlx5 driver as part of this series to
allow user space applications to accelerate traffic which is typically handled
by IPoIB ULP.

PR was sent:
https://github.com/linux-rdma/rdma-core/pull/212

Yishai Hadas (3):
  verbs: Enable creating QP with a given source QP number
  mlx5: Add support for managing QP with a given source QP number
  mlx5: Add support for sending IPoIB packets

 libibverbs/cmd.c                  |  6 +++-
 libibverbs/kern-abi.h             |  2 +-
 libibverbs/man/ibv_create_qp_ex.3 |  1 +
 libibverbs/verbs.h                |  2 ++
 providers/mlx5/mlx5.h             |  8 ++++-
 providers/mlx5/qp.c               | 68 ++++++++++++++++++++++++++++++++++++++-
 providers/mlx5/verbs.c            | 46 +++++++++++++++++++++-----
 providers/mlx5/wqe.h              |  6 ++++
 8 files changed, 127 insertions(+), 12 deletions(-)

-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-core 1/3] verbs: Enable creating QP with a given source QP number

[Yishai Hadas]

Enable QP creation with a given source QP number.
The created QP will use the source QPN as its wire QP number.

This comes as a pre-patch for downstream patches in this series to
allow user space applications to accelerate traffic which is
typically handled by IPoIB ULP.

In the IPoIB use case, the source QPN can be achieved by using some
networking tool as of ifconfig, ip link show on the interface and use
the value of 3 bytes from the hardware address starting after the first
byte.

Signed-off-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 libibverbs/cmd.c                  | 6 +++++-
 libibverbs/kern-abi.h             | 2 +-
 libibverbs/man/ibv_create_qp_ex.3 | 1 +
 libibverbs/verbs.h                | 2 ++
 4 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/libibverbs/cmd.c b/libibverbs/cmd.c
index 7a266ee..9d2140f 100644
--- a/libibverbs/cmd.c
+++ b/libibverbs/cmd.c
@@ -927,7 +927,8 @@ static void create_qp_handle_resp_common(struct ibv_context *context,
 enum {
 	CREATE_QP_EX2_SUP_CREATE_FLAGS = IBV_QP_CREATE_BLOCK_SELF_MCAST_LB |
 					 IBV_QP_CREATE_SCATTER_FCS |
-					 IBV_QP_CREATE_CVLAN_STRIPPING,
+					 IBV_QP_CREATE_CVLAN_STRIPPING |
+					 IBV_QP_CREATE_SOURCE_QPN,
 };
 
 int ibv_cmd_create_qp_ex2(struct ibv_context *context,
@@ -967,6 +968,9 @@ int ibv_cmd_create_qp_ex2(struct ibv_context *context,
 				    sizeof(qp_attr->create_flags))
 			return EINVAL;
 		cmd->create_flags = qp_attr->create_flags;
+
+		if (qp_attr->create_flags & IBV_QP_CREATE_SOURCE_QPN)
+			cmd->source_qpn = qp_attr->source_qpn;
 	}
 
 	if (qp_attr->comp_mask & IBV_QP_INIT_ATTR_IND_TABLE) {
diff --git a/libibverbs/kern-abi.h b/libibverbs/kern-abi.h
index 8d42faf..3a0e02a 100644
--- a/libibverbs/kern-abi.h
+++ b/libibverbs/kern-abi.h
@@ -679,7 +679,7 @@ struct ibv_create_qp_ex {
 	__u32 comp_mask;
 	__u32 create_flags;
 	__u32 ind_tbl_handle;
-	__u32 reserved1;
+	__u32 source_qpn;
 };
 
 struct ibv_create_qp_resp_ex {
diff --git a/libibverbs/man/ibv_create_qp_ex.3 b/libibverbs/man/ibv_create_qp_ex.3
index 99ae497..3272c09 100644
--- a/libibverbs/man/ibv_create_qp_ex.3
+++ b/libibverbs/man/ibv_create_qp_ex.3
@@ -38,6 +38,7 @@ enum ibv_qp_create_flags create_flags;	/* Creation flags for this QP */
 uint16_t                max_tso_header; /* Maximum TSO header size */
 struct ibv_rwq_ind_table *rwq_ind_tbl;  /* Indirection table to be associated with the QP */
 struct ibv_rx_hash_conf  rx_hash_conf;  /* RX hash configuration to be used */
+uint32_t                source_qpn;     /* Source QP number, creation flag IBV_QP_CREATE_SOURCE_QPN should be set */
 .in -8
 };
 .sp
diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index 922a011..3c8b2c9 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -790,6 +790,7 @@ enum ibv_qp_create_flags {
 	IBV_QP_CREATE_BLOCK_SELF_MCAST_LB	= 1 << 1,
 	IBV_QP_CREATE_SCATTER_FCS		= 1 << 8,
 	IBV_QP_CREATE_CVLAN_STRIPPING		= 1 << 9,
+	IBV_QP_CREATE_SOURCE_QPN		= 1 << 10,
 };
 
 struct ibv_rx_hash_conf {
@@ -817,6 +818,7 @@ struct ibv_qp_init_attr_ex {
 	uint16_t		max_tso_header;
 	struct ibv_rwq_ind_table       *rwq_ind_tbl;
 	struct ibv_rx_hash_conf	rx_hash_conf;
+	uint32_t		source_qpn;
 };
 
 enum ibv_qp_open_attr_mask {
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-core 2/3] mlx5: Add support for managing QP with a given source QP number

[Yishai Hadas]

Add support to create a QP with source QPN (e.g. IPoIB QPN) over UD QP.

It includes:
- Input checking that basic QP type is UD.
- Create and manage driver resources as done for RAW QP.

Signed-off-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 providers/mlx5/mlx5.h  |  8 +++++++-
 providers/mlx5/qp.c    |  3 ++-
 providers/mlx5/verbs.c | 36 +++++++++++++++++++++++++++++++-----
 3 files changed, 40 insertions(+), 7 deletions(-)

diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h
index ad36cbf..ae39d17 100644
--- a/providers/mlx5/mlx5.h
+++ b/providers/mlx5/mlx5.h
@@ -150,7 +150,8 @@ enum {
 };
 
 enum {
-	MLX5_CSUM_SUPPORT_RAW_OVER_ETH  = (1 <<  0),
+	MLX5_CSUM_SUPPORT_RAW_OVER_ETH  = (1 << 0),
+	MLX5_CSUM_SUPPORT_UNDERLAY_UD   = (1 << 1),
 	/*
 	 * Only report rx checksum when the validation
 	 * is valid.
@@ -396,6 +397,10 @@ struct mlx5_mr {
 	uint32_t			alloc_flags;
 };
 
+enum mlx5_qp_flags {
+	MLX5_QP_FLAGS_USE_UNDERLAY = 0x01,
+};
+
 struct mlx5_qp {
 	struct mlx5_resource            rsc; /* This struct must be first */
 	struct verbs_qp			verbs_qp;
@@ -421,6 +426,7 @@ struct mlx5_qp {
 	uint32_t			max_tso;
 	uint16_t			max_tso_header;
 	int                             rss_qp;
+	uint32_t			flags; /* Use enum mlx5_qp_flags */
 };
 
 struct mlx5_ah {
diff --git a/providers/mlx5/qp.c b/providers/mlx5/qp.c
index 52da8c1..2960ba0 100644
--- a/providers/mlx5/qp.c
+++ b/providers/mlx5/qp.c
@@ -1199,7 +1199,8 @@ out:
 		 * This is only for Raw Packet QPs since they are represented
 		 * differently in the hardware.
 		 */
-		if (likely(!(ibqp->qp_type == IBV_QPT_RAW_PACKET &&
+		if (likely(!((ibqp->qp_type == IBV_QPT_RAW_PACKET ||
+			      qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) &&
 			     ibqp->state < IBV_QPS_RTR)))
 			qp->db[MLX5_RCV_DBR] = htobe32(qp->rq.head & 0xffff);
 	}
diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c
index fc63ae9..a469c93 100644
--- a/providers/mlx5/verbs.c
+++ b/providers/mlx5/verbs.c
@@ -1096,7 +1096,8 @@ static int mlx5_alloc_qp_buf(struct ibv_context *context,
 
 	memset(qp->buf.buf, 0, qp->buf_size);
 
-	if (attr->qp_type == IBV_QPT_RAW_PACKET) {
+	if (attr->qp_type == IBV_QPT_RAW_PACKET ||
+	    qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) {
 		size_t aligned_sq_buf_size = align(qp->sq_buf_size,
 						   to_mdev(context->device)->page_size);
 		/* For Raw Packet QP, allocate a separate buffer for the SQ */
@@ -1257,6 +1258,17 @@ static struct ibv_qp *create_qp(struct ibv_context *context,
 	ibqp = (struct ibv_qp *)&qp->verbs_qp;
 	qp->ibv_qp = ibqp;
 
+	if ((attr->comp_mask & IBV_QP_INIT_ATTR_CREATE_FLAGS) &&
+		(attr->create_flags & IBV_QP_CREATE_SOURCE_QPN)) {
+
+		if (attr->qp_type != IBV_QPT_UD) {
+			errno = EINVAL;
+			goto err;
+		}
+
+		qp->flags |= MLX5_QP_FLAGS_USE_UNDERLAY;
+	}
+
 	memset(&cmd, 0, sizeof(cmd));
 	memset(&resp, 0, sizeof(resp));
 	memset(&resp_ex, 0, sizeof(resp_ex));
@@ -1282,7 +1294,8 @@ static struct ibv_qp *create_qp(struct ibv_context *context,
 		goto err;
 	}
 
-	if (attr->qp_type == IBV_QPT_RAW_PACKET) {
+	if (attr->qp_type == IBV_QPT_RAW_PACKET ||
+	    qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) {
 		qp->buf_size = qp->sq.offset;
 		qp->sq_buf_size = ret - qp->buf_size;
 		qp->sq.offset = 0;
@@ -1296,7 +1309,8 @@ static struct ibv_qp *create_qp(struct ibv_context *context,
 		goto err;
 	}
 
-	if (attr->qp_type == IBV_QPT_RAW_PACKET) {
+	if (attr->qp_type == IBV_QPT_RAW_PACKET ||
+	    qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) {
 		qp->sq_start = qp->sq_buf.buf;
 		qp->sq.qend = qp->sq_buf.buf +
 				(qp->sq.wqe_cnt << qp->sq.wqe_shift);
@@ -1322,7 +1336,8 @@ static struct ibv_qp *create_qp(struct ibv_context *context,
 	qp->db[MLX5_SND_DBR] = 0;
 
 	cmd.buf_addr = (uintptr_t) qp->buf.buf;
-	cmd.sq_buf_addr = (attr->qp_type == IBV_QPT_RAW_PACKET) ?
+	cmd.sq_buf_addr = (attr->qp_type == IBV_QPT_RAW_PACKET ||
+			   qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) ?
 			  (uintptr_t) qp->sq_buf.buf : 0;
 	cmd.db_addr  = (uintptr_t) qp->db;
 	cmd.sq_wqe_count = qp->sq.wqe_cnt;
@@ -1560,6 +1575,16 @@ int mlx5_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr,
 	if (mqp->rss_qp)
 		return ENOSYS;
 
+	if (mqp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) {
+		if (attr_mask & ~(IBV_QP_STATE | IBV_QP_CUR_STATE))
+			return EINVAL;
+
+		/* Underlay QP is UD over infiniband */
+		if (context->cached_device_cap_flags & IBV_DEVICE_UD_IP_CSUM)
+			mqp->qp_cap_cache |= MLX5_CSUM_SUPPORT_UNDERLAY_UD |
+					     MLX5_RX_CSUM_VALID;
+	}
+
 	if (attr_mask & IBV_QP_PORT) {
 		switch (qp->qp_type) {
 		case IBV_QPT_RAW_PACKET:
@@ -1621,7 +1646,8 @@ int mlx5_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr,
 	if (!ret &&
 	    (attr_mask & IBV_QP_STATE) &&
 	    attr->qp_state == IBV_QPS_RTR &&
-	    qp->qp_type == IBV_QPT_RAW_PACKET) {
+	    (qp->qp_type == IBV_QPT_RAW_PACKET ||
+	     mqp->flags & MLX5_QP_FLAGS_USE_UNDERLAY)) {
 		mlx5_spin_lock(&mqp->rq.lock);
 		mqp->db[MLX5_RCV_DBR] = htobe32(mqp->rq.head & 0xffff);
 		mlx5_spin_unlock(&mqp->rq.lock);
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 2/3] mlx5: Add support for managing QP with a given source QP number

[Yuval Shaia]

On Wed, Sep 06, 2017 at 05:38:48PM +0300, Yishai Hadas wrote:
> Add support to create a QP with source QPN (e.g. IPoIB QPN) over UD QP.

Sorry, i missed the kernel patches.
Is this UD limitation is for mlx5 only? Asking as i see no limitation in
the verb layer (patch 1/3).
If no, any plans for CM?
If yes, why?

> 
> It includes:
> - Input checking that basic QP type is UD.
> - Create and manage driver resources as done for RAW QP.
> 
> Signed-off-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> ---
>  providers/mlx5/mlx5.h  |  8 +++++++-
>  providers/mlx5/qp.c    |  3 ++-
>  providers/mlx5/verbs.c | 36 +++++++++++++++++++++++++++++++-----
>  3 files changed, 40 insertions(+), 7 deletions(-)
> 
> diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h
> index ad36cbf..ae39d17 100644
> --- a/providers/mlx5/mlx5.h
> +++ b/providers/mlx5/mlx5.h
> @@ -150,7 +150,8 @@ enum {
>  };
>  
>  enum {
> -	MLX5_CSUM_SUPPORT_RAW_OVER_ETH  = (1 <<  0),
> +	MLX5_CSUM_SUPPORT_RAW_OVER_ETH  = (1 << 0),
> +	MLX5_CSUM_SUPPORT_UNDERLAY_UD   = (1 << 1),
>  	/*
>  	 * Only report rx checksum when the validation
>  	 * is valid.
> @@ -396,6 +397,10 @@ struct mlx5_mr {
>  	uint32_t			alloc_flags;
>  };
>  
> +enum mlx5_qp_flags {
> +	MLX5_QP_FLAGS_USE_UNDERLAY = 0x01,
> +};
> +
>  struct mlx5_qp {
>  	struct mlx5_resource            rsc; /* This struct must be first */
>  	struct verbs_qp			verbs_qp;
> @@ -421,6 +426,7 @@ struct mlx5_qp {
>  	uint32_t			max_tso;
>  	uint16_t			max_tso_header;
>  	int                             rss_qp;
> +	uint32_t			flags; /* Use enum mlx5_qp_flags */
>  };
>  
>  struct mlx5_ah {
> diff --git a/providers/mlx5/qp.c b/providers/mlx5/qp.c
> index 52da8c1..2960ba0 100644
> --- a/providers/mlx5/qp.c
> +++ b/providers/mlx5/qp.c
> @@ -1199,7 +1199,8 @@ out:
>  		 * This is only for Raw Packet QPs since they are represented
>  		 * differently in the hardware.
>  		 */
> -		if (likely(!(ibqp->qp_type == IBV_QPT_RAW_PACKET &&
> +		if (likely(!((ibqp->qp_type == IBV_QPT_RAW_PACKET ||
> +			      qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) &&
>  			     ibqp->state < IBV_QPS_RTR)))
>  			qp->db[MLX5_RCV_DBR] = htobe32(qp->rq.head & 0xffff);
>  	}
> diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c
> index fc63ae9..a469c93 100644
> --- a/providers/mlx5/verbs.c
> +++ b/providers/mlx5/verbs.c
> @@ -1096,7 +1096,8 @@ static int mlx5_alloc_qp_buf(struct ibv_context *context,
>  
>  	memset(qp->buf.buf, 0, qp->buf_size);
>  
> -	if (attr->qp_type == IBV_QPT_RAW_PACKET) {
> +	if (attr->qp_type == IBV_QPT_RAW_PACKET ||
> +	    qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) {
>  		size_t aligned_sq_buf_size = align(qp->sq_buf_size,
>  						   to_mdev(context->device)->page_size);
>  		/* For Raw Packet QP, allocate a separate buffer for the SQ */
> @@ -1257,6 +1258,17 @@ static struct ibv_qp *create_qp(struct ibv_context *context,
>  	ibqp = (struct ibv_qp *)&qp->verbs_qp;
>  	qp->ibv_qp = ibqp;
>  
> +	if ((attr->comp_mask & IBV_QP_INIT_ATTR_CREATE_FLAGS) &&
> +		(attr->create_flags & IBV_QP_CREATE_SOURCE_QPN)) {
> +
> +		if (attr->qp_type != IBV_QPT_UD) {
> +			errno = EINVAL;
> +			goto err;
> +		}
> +
> +		qp->flags |= MLX5_QP_FLAGS_USE_UNDERLAY;
> +	}
> +
>  	memset(&cmd, 0, sizeof(cmd));
>  	memset(&resp, 0, sizeof(resp));
>  	memset(&resp_ex, 0, sizeof(resp_ex));
> @@ -1282,7 +1294,8 @@ static struct ibv_qp *create_qp(struct ibv_context *context,
>  		goto err;
>  	}
>  
> -	if (attr->qp_type == IBV_QPT_RAW_PACKET) {
> +	if (attr->qp_type == IBV_QPT_RAW_PACKET ||
> +	    qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) {
>  		qp->buf_size = qp->sq.offset;
>  		qp->sq_buf_size = ret - qp->buf_size;
>  		qp->sq.offset = 0;
> @@ -1296,7 +1309,8 @@ static struct ibv_qp *create_qp(struct ibv_context *context,
>  		goto err;
>  	}
>  
> -	if (attr->qp_type == IBV_QPT_RAW_PACKET) {
> +	if (attr->qp_type == IBV_QPT_RAW_PACKET ||
> +	    qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) {
>  		qp->sq_start = qp->sq_buf.buf;
>  		qp->sq.qend = qp->sq_buf.buf +
>  				(qp->sq.wqe_cnt << qp->sq.wqe_shift);
> @@ -1322,7 +1336,8 @@ static struct ibv_qp *create_qp(struct ibv_context *context,
>  	qp->db[MLX5_SND_DBR] = 0;
>  
>  	cmd.buf_addr = (uintptr_t) qp->buf.buf;
> -	cmd.sq_buf_addr = (attr->qp_type == IBV_QPT_RAW_PACKET) ?
> +	cmd.sq_buf_addr = (attr->qp_type == IBV_QPT_RAW_PACKET ||
> +			   qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) ?
>  			  (uintptr_t) qp->sq_buf.buf : 0;
>  	cmd.db_addr  = (uintptr_t) qp->db;
>  	cmd.sq_wqe_count = qp->sq.wqe_cnt;
> @@ -1560,6 +1575,16 @@ int mlx5_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr,
>  	if (mqp->rss_qp)
>  		return ENOSYS;
>  
> +	if (mqp->flags & MLX5_QP_FLAGS_USE_UNDERLAY) {
> +		if (attr_mask & ~(IBV_QP_STATE | IBV_QP_CUR_STATE))
> +			return EINVAL;
> +
> +		/* Underlay QP is UD over infiniband */
> +		if (context->cached_device_cap_flags & IBV_DEVICE_UD_IP_CSUM)
> +			mqp->qp_cap_cache |= MLX5_CSUM_SUPPORT_UNDERLAY_UD |
> +					     MLX5_RX_CSUM_VALID;
> +	}
> +
>  	if (attr_mask & IBV_QP_PORT) {
>  		switch (qp->qp_type) {
>  		case IBV_QPT_RAW_PACKET:
> @@ -1621,7 +1646,8 @@ int mlx5_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr,
>  	if (!ret &&
>  	    (attr_mask & IBV_QP_STATE) &&
>  	    attr->qp_state == IBV_QPS_RTR &&
> -	    qp->qp_type == IBV_QPT_RAW_PACKET) {
> +	    (qp->qp_type == IBV_QPT_RAW_PACKET ||
> +	     mqp->flags & MLX5_QP_FLAGS_USE_UNDERLAY)) {
>  		mlx5_spin_lock(&mqp->rq.lock);
>  		mqp->db[MLX5_RCV_DBR] = htobe32(mqp->rq.head & 0xffff);
>  		mlx5_spin_unlock(&mqp->rq.lock);
> -- 
> 1.8.3.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 2/3] mlx5: Add support for managing QP with a given source QP number

[Yishai Hadas]

On 9/6/2017 6:12 PM, Yuval Shaia wrote:
> On Wed, Sep 06, 2017 at 05:38:48PM +0300, Yishai Hadas wrote:
>> Add support to create a QP with source QPN (e.g. IPoIB QPN) over UD QP.
>
> Sorry, i missed the kernel patches.
> Is this UD limitation is for mlx5 only? Asking as i see no limitation in
> the verb layer (patch 1/3).

 From API point of view as was introduced by the verbs patch there is no 
limitation. The mlx5 driver implements the use case for UD QP to 
accelerate IPoIB packets.

> If no, any plans for CM?
No plan for CM at the moment, however, it may be considered in the 
future based on demand/use case.

Thanks for looking at.
Yishai
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 2/3] mlx5: Add support for managing QP with a given source QP number

[Yuval Shaia]

On Wed, Sep 06, 2017 at 06:56:29PM +0300, Yishai Hadas wrote:
> On 9/6/2017 6:12 PM, Yuval Shaia wrote:
> >On Wed, Sep 06, 2017 at 05:38:48PM +0300, Yishai Hadas wrote:
> >>Add support to create a QP with source QPN (e.g. IPoIB QPN) over UD QP.
> >
> >Sorry, i missed the kernel patches.
> >Is this UD limitation is for mlx5 only? Asking as i see no limitation in
> >the verb layer (patch 1/3).
> 
> From API point of view as was introduced by the verbs patch there is
> no limitation. The mlx5 driver implements the use case for UD QP to
> accelerate IPoIB packets.
> 
> >If no, any plans for CM?
> No plan for CM at the moment, however, it may be considered in the
> future based on demand/use case.

I can give you use case - migration.

> 
> Thanks for looking at.
> Yishai
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 2/3] mlx5: Add support for managing QP with a given source QP number

[Jason Gunthorpe]

On Wed, Sep 06, 2017 at 11:50:15PM -0700, Yuval Shaia wrote:
> On Wed, Sep 06, 2017 at 06:56:29PM +0300, Yishai Hadas wrote:
> > On 9/6/2017 6:12 PM, Yuval Shaia wrote:
> > >On Wed, Sep 06, 2017 at 05:38:48PM +0300, Yishai Hadas wrote:
> > >>Add support to create a QP with source QPN (e.g. IPoIB QPN) over UD QP.
> > >
> > >Sorry, i missed the kernel patches.
> > >Is this UD limitation is for mlx5 only? Asking as i see no limitation in
> > >the verb layer (patch 1/3).
> > 
> > From API point of view as was introduced by the verbs patch there is
> > no limitation. The mlx5 driver implements the use case for UD QP to
> > accelerate IPoIB packets.
> > 
> > >If no, any plans for CM?
> > No plan for CM at the moment, however, it may be considered in the
> > future based on demand/use case.
> 
> I can give you use case - migration.

Maybe Yishai should adjust the name and description for this feature.

It creates a SEND ONLY QP that is non-exclusive to normal UD QPs using
the same QPN.

What you are asking for is a way to create a new QP of any type with a
fixed QPN so long as the QPN does not overlap with an existing
QPN.. Very different.

.. and migration needs a way to import quite alot of other state too,
so not sure a fixed QPN is really useful for that in any event.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-core 3/3] mlx5: Add support for sending IPoIB packets

[Yishai Hadas]

It includes:
- Update WQE size to match the addition required segments.
- Build a proper IPoIB packet as part of the post send flow.

Signed-off-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 providers/mlx5/qp.c    | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++
 providers/mlx5/verbs.c | 10 +++++---
 providers/mlx5/wqe.h   |  6 +++++
 3 files changed, 78 insertions(+), 3 deletions(-)

diff --git a/providers/mlx5/qp.c b/providers/mlx5/qp.c
index 2960ba0..7b4a754 100644
--- a/providers/mlx5/qp.c
+++ b/providers/mlx5/qp.c
@@ -604,6 +604,63 @@ static inline int set_tso_eth_seg(void **seg, struct ibv_send_wr *wr,
 	return 0;
 }
 
+static inline int mlx5_post_send_underlay(struct mlx5_qp *qp, struct ibv_send_wr *wr,
+					  void **pseg, int *total_size,
+					  struct mlx5_sg_copy_ptr *sg_copy_ptr)
+{
+	struct mlx5_wqe_eth_seg *eseg;
+	int inl_hdr_copy_size;
+	void *seg = *pseg;
+	int size = 0;
+
+	if (unlikely(wr->opcode == IBV_WR_SEND_WITH_IMM))
+		return EINVAL;
+
+	memset(seg, 0, sizeof(struct mlx5_wqe_eth_pad));
+	size += sizeof(struct mlx5_wqe_eth_pad);
+	seg += sizeof(struct mlx5_wqe_eth_pad);
+	eseg = seg;
+	*((uint64_t *)eseg) = 0;
+	eseg->rsvd2 = 0;
+
+	if (wr->send_flags & IBV_SEND_IP_CSUM) {
+		if (!(qp->qp_cap_cache & MLX5_CSUM_SUPPORT_UNDERLAY_UD))
+			return EINVAL;
+
+		eseg->cs_flags |= MLX5_ETH_WQE_L3_CSUM | MLX5_ETH_WQE_L4_CSUM;
+	}
+
+	if (likely(wr->sg_list[0].length >= MLX5_IPOIB_INLINE_MAX_HEADER_SIZE))
+		/* Copying the minimum required data unless inline mode is set */
+		inl_hdr_copy_size = (wr->send_flags & IBV_SEND_INLINE) ?
+				MLX5_IPOIB_INLINE_MAX_HEADER_SIZE :
+				MLX5_IPOIB_INLINE_MIN_HEADER_SIZE;
+	else {
+		inl_hdr_copy_size = MLX5_IPOIB_INLINE_MIN_HEADER_SIZE;
+		/* We expect at least 4 bytes as part of first entry to hold the IPoIB header */
+		if (unlikely(wr->sg_list[0].length < inl_hdr_copy_size))
+			return EINVAL;
+	}
+
+	memcpy(eseg->inline_hdr_start, (void *)(uintptr_t)wr->sg_list[0].addr,
+	       inl_hdr_copy_size);
+	eseg->inline_hdr_sz = htobe16(inl_hdr_copy_size);
+	size += sizeof(struct mlx5_wqe_eth_seg);
+	seg += sizeof(struct mlx5_wqe_eth_seg);
+
+	/* If we copied all the sge into the inline-headers, then we need to
+	 * start copying from the next sge into the data-segment.
+	 */
+	if (unlikely(wr->sg_list[0].length == inl_hdr_copy_size))
+		sg_copy_ptr->index++;
+	else
+		sg_copy_ptr->offset = inl_hdr_copy_size;
+
+	*pseg = seg;
+	*total_size += (size / 16);
+	return 0;
+}
+
 static inline int _mlx5_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr,
 				  struct ibv_send_wr **bad_wr)
 {
@@ -806,6 +863,14 @@ static inline int _mlx5_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr,
 			size += sizeof(struct mlx5_wqe_datagram_seg) / 16;
 			if (unlikely((seg == qend)))
 				seg = mlx5_get_send_wqe(qp, 0);
+
+			if (unlikely(qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY)) {
+				err = mlx5_post_send_underlay(qp, wr, &seg, &size, &sg_copy_ptr);
+				if (unlikely(err)) {
+					*bad_wr = wr;
+					goto out;
+				}
+			}
 			break;
 
 		case IBV_QPT_RAW_PACKET:
diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c
index a469c93..5a360a9 100644
--- a/providers/mlx5/verbs.c
+++ b/providers/mlx5/verbs.c
@@ -756,7 +756,7 @@ int mlx5_destroy_srq(struct ibv_srq *srq)
 	return 0;
 }
 
-static int sq_overhead(enum ibv_qp_type	qp_type)
+static int sq_overhead(struct mlx5_qp *qp, enum ibv_qp_type qp_type)
 {
 	size_t size = 0;
 	size_t mw_bind_size =
@@ -781,6 +781,10 @@ static int sq_overhead(enum ibv_qp_type	qp_type)
 	case IBV_QPT_UD:
 		size = sizeof(struct mlx5_wqe_ctrl_seg) +
 			sizeof(struct mlx5_wqe_datagram_seg);
+
+		if (qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY)
+			size += (sizeof(struct mlx5_wqe_eth_seg) + sizeof(struct mlx5_wqe_eth_pad));
+
 		break;
 
 	case IBV_QPT_XRC_SEND:
@@ -814,7 +818,7 @@ static int mlx5_calc_send_wqe(struct mlx5_context *ctx,
 	int max_gather;
 	int tot_size;
 
-	size = sq_overhead(attr->qp_type);
+	size = sq_overhead(qp, attr->qp_type);
 	if (size < 0)
 		return size;
 
@@ -887,7 +891,7 @@ static int mlx5_calc_sq_size(struct mlx5_context *ctx,
 		return -EINVAL;
 	}
 
-	qp->max_inline_data = wqe_size - sq_overhead(attr->qp_type) -
+	qp->max_inline_data = wqe_size - sq_overhead(qp, attr->qp_type) -
 		sizeof(struct mlx5_wqe_inl_data_seg);
 	attr->cap.max_inline_data = qp->max_inline_data;
 
diff --git a/providers/mlx5/wqe.h b/providers/mlx5/wqe.h
index 063dc9a..f3f7964 100644
--- a/providers/mlx5/wqe.h
+++ b/providers/mlx5/wqe.h
@@ -50,6 +50,10 @@ struct mlx5_eqe_qp_srq {
 	uint32_t	qp_srq_n;
 };
 
+struct mlx5_wqe_eth_pad {
+	uint8_t rsvd0[16];
+};
+
 struct mlx5_wqe_xrc_seg {
 	__be32		xrc_srqn;
 	uint8_t		rsvd[12];
@@ -63,6 +67,8 @@ struct mlx5_wqe_masked_atomic_seg {
 };
 
 enum {
+	MLX5_IPOIB_INLINE_MIN_HEADER_SIZE	= 4,
+	MLX5_IPOIB_INLINE_MAX_HEADER_SIZE	= 18,
 	MLX5_ETH_L2_INLINE_HEADER_SIZE	= 18,
 	MLX5_ETH_L2_MIN_HEADER_SIZE	= 14,
 };
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 3/3] mlx5: Add support for sending IPoIB packets

[Jason Gunthorpe]

On Wed, Sep 06, 2017 at 05:38:49PM +0300, Yishai Hadas wrote:

> +			if (unlikely(qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY)) {
> +				err = mlx5_post_send_underlay(qp, wr, &seg, &size, &sg_copy_ptr);
> +				if (unlikely(err)) {
> +					*bad_wr = wr;
> +					goto out;
> +				}
> +			}

I don't know anything about your device, but this patch looks to me
like it enables for any IBV_QP_CREATE_SOURCE_QPN user - and it seems
to contain a lot of IPoIB specific commentary.

It is wrong to assume that all IBV_QP_CREATE_SOURCE_QPN QP's are IPoIB.

At a minimum some of the commentary needs revising..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 3/3] mlx5: Add support for sending IPoIB packets

[Yishai Hadas]

On 9/6/2017 7:04 PM, Jason Gunthorpe wrote:
> On Wed, Sep 06, 2017 at 05:38:49PM +0300, Yishai Hadas wrote:
>
>> +			if (unlikely(qp->flags & MLX5_QP_FLAGS_USE_UNDERLAY)) {
>> +				err = mlx5_post_send_underlay(qp, wr, &seg, &size, &sg_copy_ptr);
>> +				if (unlikely(err)) {
>> +					*bad_wr = wr;
>> +					goto out;
>> +				}
>> +			}
>
> I don't know anything about your device, but this patch looks to me
> like it enables for any IBV_QP_CREATE_SOURCE_QPN user

Correct, the code is generic and can work not only with IPoIB packets.

  - and it seems
> to contain a lot of IPoIB specific commentary.
>
> It is wrong to assume that all IBV_QP_CREATE_SOURCE_QPN QP's are IPoIB.
>
> At a minimum some of the commentary needs revising..

The commentary was focused on IPoIB as this is the main use case that we 
come to solve.

I'll revise the commit log to better describe the general case and 
mention the IPoIB as a specific use case, will send V1 for that.

Thanks for comment on.
Yishai

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 3/3] mlx5: Add support for sending IPoIB packets

[Jason Gunthorpe]

On Wed, Sep 06, 2017 at 07:49:03PM +0300, Yishai Hadas wrote:

> The commentary was focused on IPoIB as this is the main use case that we
> come to solve.

It is more than that, why does mlx5_post_send_underlay refer to
things like MLX5_IPOIB_INLINE_MAX_HEADER_SIZE ? Those constants seem
mis-named.

And then there are comments like this:

> /* We expect at least 4 bytes as part of first entry to hold the IPoIB header */

Which really doesn't sound general at all.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 3/3] mlx5: Add support for sending IPoIB packets

[Yishai Hadas]

On 9/6/2017 7:53 PM, Jason Gunthorpe wrote:
> On Wed, Sep 06, 2017 at 07:49:03PM +0300, Yishai Hadas wrote:
>
>> The commentary was focused on IPoIB as this is the main use case that we
>> come to solve.
>
> It is more than that, why does mlx5_post_send_underlay refer to
> things like MLX5_IPOIB_INLINE_MAX_HEADER_SIZE ? Those constants seem
> mis-named.
>
> And then there are comments like this:
>
>> /* We expect at least 4 bytes as part of first entry to hold the IPoIB header */
>
> Which really doesn't sound general at all.

The specific IPoIB handling as of copying from the application gather 
data into the header is some device limitation to support sending IPoIB 
packets. This is not required in the general case but will work as well. 
Future devices may work properly without that need, once be ready we may 
have some device capability exposed from kernel and clean this specific 
handling. Will update commit log to include this information as well.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html