[PATCH rdma-next 0/2] Introduce Multi-Packet receive work queue

[PATCH rdma-next 0/2] Introduce Multi-Packet receive work queue

[Leon Romanovsky]

A Multi-Packet receive WQ comes to improve performance and memory
footprint. It is comprised of work requests that can be consumed by
multiple packets. Buffers pointed by SGEs of each work request in a
multi packet WQ are divided into strides. Both buffer size and stride
size must be identical in all work requests of the same multi packet
WQ and are set during WQ creation.

A single WR in a Multi-Packet WQ can generate multiple completions,
depends on the number of packets consumed. Each packet starts at the
beginning of a stride and is written in stride granularity.

Here is an example of a work request comprised of 3 strides. Packet 1
is larger than a stride so it occupies 2 strides. Packet 2 fits in a
single stride. Each packet will generate its own completion.

|         |         |[packet2]|
|    [packet1]      |         |
|_________|_________|_________|
 stride      stride     stride
 <--      work request     -->

A Multi-Packet WQ reduces the number of needed post-recv operations
thus increasing performance: In the example above, a single post-recv
operation was performed instead of 2.

It reduces memory footprint by allowing each packet to consume a
different number of strides instead of the whole WR. For example, a
large work request with many small strides: If the incoming packet is
smaller than a stride, the waste will be <stride size> - <packet size>
while for a large packet the waste will be:
<stride size> - (<packet size> % <stride size>)

The patches are available in the git repository at:
  git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git tags/rdma-next-2017-10-17

	Thanks
---------------------------------------

Noa Osherovich (2):
  IB/mlx5: Expose multi-packet RQ capabilities
  IB/mlx5: Allow creation of a multi-packet RQ

 drivers/infiniband/hw/mlx5/main.c    | 16 +++++++++++
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  9 +++++++
 drivers/infiniband/hw/mlx5/qp.c      | 52 ++++++++++++++++++++++++++++++------
 include/linux/mlx5/mlx5_ifc.h        |  1 +
 include/uapi/rdma/mlx5-abi.h         | 22 ++++++++++++++-
 5 files changed, 91 insertions(+), 9 deletions(-)

--
2.14.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 1/2] IB/mlx5: Expose multi-packet RQ capabilities

[Leon Romanovsky]

From: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

This patch reports the device's striding RQ capabilities to
the user-space:
- min/max_single_stride_log_num_of_bytes: Log of min/max number of
  bytes in a single stride.
- min/max_single_wqe_log_num_of_strides: Log of min/max number of
  strides in a single WQE.
- supported_qpts: A bit mask to know which QP types support multi-
  packet RQ, for now only Raw Packet QPs.

Signed-off-by: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/hw/mlx5/main.c    | 16 ++++++++++++++++
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  5 +++++
 include/uapi/rdma/mlx5-abi.h         | 14 ++++++++++++++
 3 files changed, 35 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 552f7bd4ecc3..02da3f58f296 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -848,6 +848,22 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 		}
 	}
 
+	if (field_avail(typeof(resp), striding_rq_caps, uhw->outlen)) {
+		resp.response_length += sizeof(resp.striding_rq_caps);
+		if (MLX5_CAP_GEN(mdev, striding_rq)) {
+			resp.striding_rq_caps.min_single_stride_log_num_of_bytes =
+				MLX5_MIN_SINGLE_STRIDE_LOG_NUM_BYTES;
+			resp.striding_rq_caps.max_single_stride_log_num_of_bytes =
+				MLX5_MAX_SINGLE_STRIDE_LOG_NUM_BYTES;
+			resp.striding_rq_caps.min_single_wqe_log_num_of_strides =
+				MLX5_MIN_SINGLE_WQE_LOG_NUM_STRIDES;
+			resp.striding_rq_caps.max_single_wqe_log_num_of_strides =
+				MLX5_MAX_SINGLE_WQE_LOG_NUM_STRIDES;
+			resp.striding_rq_caps.supported_qpts =
+				BIT(IB_QPT_RAW_PACKET);
+		}
+	}
+
 	if (uhw->outlen) {
 		err = ib_copy_to_udata(uhw, &resp, resp.response_length);
 
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 189e80cd6b2f..8de40852818b 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -256,6 +256,11 @@ enum mlx5_ib_wq_flags {
 	MLX5_IB_WQ_FLAGS_DELAY_DROP = 0x1,
 };
 
+#define MLX5_MIN_SINGLE_WQE_LOG_NUM_STRIDES 9
+#define MLX5_MAX_SINGLE_WQE_LOG_NUM_STRIDES 16
+#define MLX5_MIN_SINGLE_STRIDE_LOG_NUM_BYTES 6
+#define MLX5_MAX_SINGLE_STRIDE_LOG_NUM_BYTES 13
+
 struct mlx5_ib_rwq {
 	struct ib_wq		ibwq;
 	struct mlx5_core_qp	core_qp;
diff --git a/include/uapi/rdma/mlx5-abi.h b/include/uapi/rdma/mlx5-abi.h
index 1791bf123ba9..0832d9502200 100644
--- a/include/uapi/rdma/mlx5-abi.h
+++ b/include/uapi/rdma/mlx5-abi.h
@@ -190,6 +190,19 @@ struct mlx5_ib_sw_parsing_caps {
 	__u32 supported_qpts;
 };
 
+struct mlx5_ib_striding_rq_caps {
+	__u32 min_single_stride_log_num_of_bytes;
+	__u32 max_single_stride_log_num_of_bytes;
+	__u32 min_single_wqe_log_num_of_strides;
+	__u32 max_single_wqe_log_num_of_strides;
+
+	/* Corresponding bit will be set if qp type from
+	 * 'enum ib_qp_type' is supported, e.g.
+	 * supported_qpts |= 1 << IB_QPT_RAW_PACKET
+	 */
+	__u32 supported_qpts;
+};
+
 struct mlx5_ib_query_device_resp {
 	__u32	comp_mask;
 	__u32	response_length;
@@ -200,6 +213,7 @@ struct mlx5_ib_query_device_resp {
 	__u32	mlx5_ib_support_multi_pkt_send_wqes;
 	__u32	reserved;
 	struct mlx5_ib_sw_parsing_caps sw_parsing_caps;
+	struct mlx5_ib_striding_rq_caps striding_rq_caps;
 };
 
 struct mlx5_ib_create_cq {
-- 
2.14.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 2/2] IB/mlx5: Allow creation of a multi-packet RQ

[Leon Romanovsky]

From: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Allow creation of a multi-packet receive queue.

In order to create a multi-packet RQ, the following fields in
the mlx5_ib_rwq should be set:
- log_num_strides: Log of number of strides per WQE
- single_stride_log_num_of_bytes: Log of a single stride size
- two_byte_shift_en: When enabled, hardware pads 2 bytes of zeros
  before writing the message to memory (e.g. for the IP alignment).

Signed-off-by: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  4 +++
 drivers/infiniband/hw/mlx5/qp.c      | 52 ++++++++++++++++++++++++++++++------
 include/linux/mlx5/mlx5_ifc.h        |  1 +
 include/uapi/rdma/mlx5-abi.h         |  8 +++++-
 4 files changed, 56 insertions(+), 9 deletions(-)

diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 8de40852818b..e7deaa08535b 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -254,6 +254,7 @@ struct mlx5_ib_wq {
 
 enum mlx5_ib_wq_flags {
 	MLX5_IB_WQ_FLAGS_DELAY_DROP = 0x1,
+	MLX5_IB_WQ_FLAGS_STRIDING_RQ = 0x2,
 };
 
 #define MLX5_MIN_SINGLE_WQE_LOG_NUM_STRIDES 9
@@ -269,6 +270,9 @@ struct mlx5_ib_rwq {
 	u32			log_rq_size;
 	u32			rq_page_offset;
 	u32			log_page_size;
+	u32			log_num_strides;
+	u32			two_byte_shift_en;
+	u32			single_stride_log_num_of_bytes;
 	struct ib_umem		*umem;
 	size_t			buf_size;
 	unsigned int		page_shift;
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 37a0976240fd..d209c684d729 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -4706,9 +4706,19 @@ static int  create_rq(struct mlx5_ib_rwq *rwq, struct ib_pd *pd,
 	MLX5_SET(rqc,  rqc, state, MLX5_RQC_STATE_RST);
 	MLX5_SET(rqc,  rqc, flush_in_error_en, 1);
 	wq = MLX5_ADDR_OF(rqc, rqc, wq);
-	MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_CYCLIC);
+	MLX5_SET(wq, wq, wq_type,
+		 rwq->create_flags & MLX5_IB_WQ_FLAGS_STRIDING_RQ ?
+		 MLX5_WQ_TYPE_CYCLIC_STRIDING_RQ : MLX5_WQ_TYPE_CYCLIC);
 	MLX5_SET(wq, wq, end_padding_mode, MLX5_WQ_END_PAD_MODE_ALIGN);
 	MLX5_SET(wq, wq, log_wq_stride, rwq->log_rq_stride);
+	if (rwq->create_flags & MLX5_IB_WQ_FLAGS_STRIDING_RQ) {
+		MLX5_SET(wq, wq, two_byte_shift_en, rwq->two_byte_shift_en);
+		MLX5_SET(wq, wq, log_wqe_stride_size,
+			 rwq->single_stride_log_num_of_bytes -
+			 MLX5_MIN_SINGLE_STRIDE_LOG_NUM_BYTES);
+		MLX5_SET(wq, wq, log_wqe_num_of_strides, rwq->log_num_strides -
+			 MLX5_MIN_SINGLE_WQE_LOG_NUM_STRIDES);
+	}
 	MLX5_SET(wq, wq, log_wq_sz, rwq->log_rq_size);
 	MLX5_SET(wq, wq, pd, to_mpd(pd)->pdn);
 	MLX5_SET(wq, wq, page_offset, rwq->rq_page_offset);
@@ -4790,7 +4800,8 @@ static int prepare_user_rq(struct ib_pd *pd,
 	int err;
 	size_t required_cmd_sz;
 
-	required_cmd_sz = offsetof(typeof(ucmd), reserved) + sizeof(ucmd.reserved);
+	required_cmd_sz = offsetof(typeof(ucmd), single_stride_log_num_of_bytes)
+		+ sizeof(ucmd.single_stride_log_num_of_bytes);
 	if (udata->inlen < required_cmd_sz) {
 		mlx5_ib_dbg(dev, "invalid inlen\n");
 		return -EINVAL;
@@ -4808,14 +4819,39 @@ static int prepare_user_rq(struct ib_pd *pd,
 		return -EFAULT;
 	}
 
-	if (ucmd.comp_mask) {
+	if (ucmd.comp_mask & (~MLX5_IB_CREATE_WQ_STRIDING_RQ)) {
 		mlx5_ib_dbg(dev, "invalid comp mask\n");
 		return -EOPNOTSUPP;
-	}
-
-	if (ucmd.reserved) {
-		mlx5_ib_dbg(dev, "invalid reserved\n");
-		return -EOPNOTSUPP;
+	} else if (ucmd.comp_mask & MLX5_IB_CREATE_WQ_STRIDING_RQ) {
+		if (!MLX5_CAP_GEN(dev->mdev, striding_rq)) {
+			mlx5_ib_dbg(dev, "Striding RQ is not supported\n");
+			return -EOPNOTSUPP;
+		}
+		if ((ucmd.single_stride_log_num_of_bytes <
+		    MLX5_MIN_SINGLE_STRIDE_LOG_NUM_BYTES) ||
+		    (ucmd.single_stride_log_num_of_bytes >
+		     MLX5_MAX_SINGLE_STRIDE_LOG_NUM_BYTES)) {
+			mlx5_ib_dbg(dev, "Invalid log stride size (%u. Range is %u - %u)\n",
+				    ucmd.single_stride_log_num_of_bytes,
+				    MLX5_MIN_SINGLE_STRIDE_LOG_NUM_BYTES,
+				    MLX5_MAX_SINGLE_STRIDE_LOG_NUM_BYTES);
+			return -EINVAL;
+		}
+		if ((ucmd.single_wqe_log_num_of_strides >
+		    MLX5_MAX_SINGLE_WQE_LOG_NUM_STRIDES) ||
+		     (ucmd.single_wqe_log_num_of_strides <
+			MLX5_MIN_SINGLE_WQE_LOG_NUM_STRIDES)) {
+			mlx5_ib_dbg(dev, "Invalid log num strides (%u. Range is %u - %u)\n",
+				    ucmd.single_wqe_log_num_of_strides,
+				    MLX5_MIN_SINGLE_WQE_LOG_NUM_STRIDES,
+				    MLX5_MAX_SINGLE_WQE_LOG_NUM_STRIDES);
+			return -EINVAL;
+		}
+		rwq->single_stride_log_num_of_bytes =
+			ucmd.single_stride_log_num_of_bytes;
+		rwq->log_num_strides = ucmd.single_wqe_log_num_of_strides;
+		rwq->two_byte_shift_en = !!ucmd.two_byte_shift_en;
+		rwq->create_flags |= MLX5_IB_WQ_FLAGS_STRIDING_RQ;
 	}
 
 	err = set_user_rq_size(dev, init_attr, &ucmd, rwq);
diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 69772347f866..db655db45b77 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -744,6 +744,7 @@ enum {
 	MLX5_WQ_TYPE_LINKED_LIST  = 0x0,
 	MLX5_WQ_TYPE_CYCLIC       = 0x1,
 	MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ = 0x2,
+	MLX5_WQ_TYPE_CYCLIC_STRIDING_RQ = 0x3,
 };
 
 enum {
diff --git a/include/uapi/rdma/mlx5-abi.h b/include/uapi/rdma/mlx5-abi.h
index 0832d9502200..b1d5b87ba3fd 100644
--- a/include/uapi/rdma/mlx5-abi.h
+++ b/include/uapi/rdma/mlx5-abi.h
@@ -308,6 +308,10 @@ struct mlx5_ib_alloc_mw {
 	__u16	reserved2;
 };
 
+enum mlx5_ib_create_wq_mask {
+	MLX5_IB_CREATE_WQ_STRIDING_RQ	= (1 << 0),
+};
+
 struct mlx5_ib_create_wq {
 	__u64   buf_addr;
 	__u64   db_addr;
@@ -316,7 +320,9 @@ struct mlx5_ib_create_wq {
 	__u32   user_index;
 	__u32   flags;
 	__u32   comp_mask;
-	__u32   reserved;
+	__u32	single_stride_log_num_of_bytes;
+	__u32	single_wqe_log_num_of_strides;
+	__u32	two_byte_shift_en;
 };
 
 struct mlx5_ib_create_ah_resp {
-- 
2.14.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 0/2] Introduce Multi-Packet receive work queue

[Sagi Grimberg]

> A Multi-Packet receive WQ comes to improve performance and memory
> footprint. It is comprised of work requests that can be consumed by
> multiple packets. Buffers pointed by SGEs of each work request in a
> multi packet WQ are divided into strides. Both buffer size and stride
> size must be identical in all work requests of the same multi packet
> WQ and are set during WQ creation.
> 
> A single WR in a Multi-Packet WQ can generate multiple completions,
> depends on the number of packets consumed. Each packet starts at the
> beginning of a stride and is written in stride granularity.
> 
> Here is an example of a work request comprised of 3 strides. Packet 1
> is larger than a stride so it occupies 2 strides. Packet 2 fits in a
> single stride. Each packet will generate its own completion.
> 
> |         |         |[packet2]|
> |    [packet1]      |         |
> |_________|_________|_________|
>   stride      stride     stride
>   <--      work request     -->
> 
> A Multi-Packet WQ reduces the number of needed post-recv operations
> thus increasing performance: In the example above, a single post-recv
> operation was performed instead of 2.

Looks like a nice feature, would be nice if we could use it... The
problem is that adding a different logic in case this feature exists is
messy and difficult. Is there any way we can come up with a verbs API
that will utilize this feature but hide the implementation details from
the ULPs?

Maybe something that wraps post_recv?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 0/2] Introduce Multi-Packet receive work queue

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1784 bytes --]

On Wed, Oct 18, 2017 at 12:55:21PM +0300, Sagi Grimberg wrote:
>
> > A Multi-Packet receive WQ comes to improve performance and memory
> > footprint. It is comprised of work requests that can be consumed by
> > multiple packets. Buffers pointed by SGEs of each work request in a
> > multi packet WQ are divided into strides. Both buffer size and stride
> > size must be identical in all work requests of the same multi packet
> > WQ and are set during WQ creation.
> >
> > A single WR in a Multi-Packet WQ can generate multiple completions,
> > depends on the number of packets consumed. Each packet starts at the
> > beginning of a stride and is written in stride granularity.
> >
> > Here is an example of a work request comprised of 3 strides. Packet 1
> > is larger than a stride so it occupies 2 strides. Packet 2 fits in a
> > single stride. Each packet will generate its own completion.
> >
> > |         |         |[packet2]|
> > |    [packet1]      |         |
> > |_________|_________|_________|
> >   stride      stride     stride
> >   <--      work request     -->
> >
> > A Multi-Packet WQ reduces the number of needed post-recv operations
> > thus increasing performance: In the example above, a single post-recv
> > operation was performed instead of 2.
>
> Looks like a nice feature, would be nice if we could use it... The
> problem is that adding a different logic in case this feature exists is
> messy and difficult. Is there any way we can come up with a verbs API
> that will utilize this feature but hide the implementation details from
> the ULPs?

We will be more than happy to work with the community and update our
driver's code if needed, once the community will converge on the proper API.
Right now, we are exposing it through our DV interface.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 0/2] Introduce Multi-Packet receive work queue

[Doug Ledford]

On Tue, 2017-10-17 at 18:01 +0300, Leon Romanovsky wrote:
> A Multi-Packet receive WQ comes to improve performance and memory
> footprint. It is comprised of work requests that can be consumed by
> multiple packets. Buffers pointed by SGEs of each work request in a
> multi packet WQ are divided into strides. Both buffer size and stride
> size must be identical in all work requests of the same multi packet
> WQ and are set during WQ creation.
> 
> A single WR in a Multi-Packet WQ can generate multiple completions,
> depends on the number of packets consumed. Each packet starts at the
> beginning of a stride and is written in stride granularity.
> 
> Here is an example of a work request comprised of 3 strides. Packet 1
> is larger than a stride so it occupies 2 strides. Packet 2 fits in a
> single stride. Each packet will generate its own completion.
> 
> >          |         |[packet2]|
> >     [packet1]      |         |
> > _________|_________|_________|
> 
>  stride      stride     stride
>  <--      work request     -->
> 
> A Multi-Packet WQ reduces the number of needed post-recv operations
> thus increasing performance: In the example above, a single post-recv
> operation was performed instead of 2.

Nice.  Resolves some of the issues the basic recv queue model has.  As
you said to Sagi, this is just being exposed in your DV interface right
now, so I'm fine with it.  Thanks, applied.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html