[PATCH rdma-next 0/3] Enable IB out-of-order by default in mlx5

[PATCH rdma-next 0/3] Enable IB out-of-order by default in mlx5

[Leon Romanovsky]

From: Leon Romanovsky <leonro@nvidia.com>

Hi,

This series from Or changes default of IB out-of-order feature and
allows to the RDMA users to decide if they need to wait for completion
for all segments or it is enough to wait for last segment completion only.

Thanks

Or Har-Toov (3):
  net/mlx5: Expose bits for enabling out-of-order by default
  RDMA/mlx5: Disable out-of-order in integrity enabled QPs
  net/mlx5: Set out of order (ooo) by default

 drivers/infiniband/hw/mlx5/qp.c                |  8 ++++++++
 drivers/net/ethernet/mellanox/mlx5/core/main.c |  3 +++
 include/linux/mlx5/mlx5_ifc.h                  | 10 +++++++---
 3 files changed, 18 insertions(+), 3 deletions(-)

-- 
2.39.2

[PATCH mlx5-next 1/3] net/mlx5: Expose bits for enabling out-of-order by default

[Leon Romanovsky]

From: Or Har-Toov <ohartoov@nvidia.com>

Add needed HW bits for enabling out-of-order by default and
use go_back_n when out-of-order is not needed.

Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 include/linux/mlx5/mlx5_ifc.h | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 15a850f52ef2..e4306cd87cd7 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -1077,7 +1077,9 @@ struct mlx5_ifc_roce_cap_bits {
 	u8         sw_r_roce_src_udp_port[0x1];
 	u8         fl_rc_qp_when_roce_disabled[0x1];
 	u8         fl_rc_qp_when_roce_enabled[0x1];
-	u8         reserved_at_7[0x17];
+	u8         reserved_at_7[0x1];
+	u8	   qp_ooo_transmit_default[0x1];
+	u8         reserved_at_9[0x15];
 	u8	   qp_ts_format[0x2];
 
 	u8         reserved_at_20[0x60];
@@ -1493,7 +1495,8 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8         reserved_at_b0[0x1];
 	u8         uplink_follow[0x1];
 	u8         ts_cqe_to_dest_cqn[0x1];
-	u8         reserved_at_b3[0x7];
+	u8         reserved_at_b3[0x6];
+	u8         go_back_n[0x1];
 	u8         shampo[0x1];
 	u8         reserved_at_bb[0x5];
 
@@ -3263,7 +3266,8 @@ struct mlx5_ifc_qpc_bits {
 	u8         log_rq_stride[0x3];
 	u8         no_sq[0x1];
 	u8         log_sq_size[0x4];
-	u8         reserved_at_55[0x3];
+	u8         reserved_at_55[0x1];
+	u8	   retry_mode[0x2];
 	u8	   ts_format[0x2];
 	u8         reserved_at_5a[0x1];
 	u8         rlky[0x1];
-- 
2.39.2

[PATCH rdma-next 2/3] RDMA/mlx5: Disable out-of-order in integrity enabled QPs

[Leon Romanovsky]

From: Or Har-Toov <ohartoov@nvidia.com>

Set retry_mode to GO_BACK_N when qp is created with INTEGRITY_EN flag
because out-of-order is not supported when doing HW offload of signature
operations.

Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 drivers/infiniband/hw/mlx5/qp.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index f04adc18e63b..d32b644885b3 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -60,6 +60,10 @@ enum raw_qp_set_mask_map {
 	MLX5_RAW_QP_RATE_LIMIT			= 1UL << 1,
 };
 
+enum {
+	MLX5_QP_RM_GO_BACK_N			= 0x1,
+};
+
 struct mlx5_modify_raw_qp_param {
 	u16 operation;
 
@@ -2521,6 +2525,10 @@ static int create_kernel_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd,
 	if (qp->flags & IB_QP_CREATE_IPOIB_UD_LSO)
 		MLX5_SET(qpc, qpc, ulp_stateless_offload_mode, 1);
 
+	if (qp->flags & IB_QP_CREATE_INTEGRITY_EN &&
+	    MLX5_CAP_GEN(mdev, go_back_n))
+		MLX5_SET(qpc, qpc, retry_mode, MLX5_QP_RM_GO_BACK_N);
+
 	err = mlx5_qpc_create_qp(dev, &base->mqp, in, inlen, out);
 	kvfree(in);
 	if (err)
-- 
2.39.2

[PATCH mlx5-next 3/3] net/mlx5: Set out of order (ooo) by default

[Leon Romanovsky]

From: Or Har-Toov <ohartoov@nvidia.com>

When FW supports ooo by default, enable the cap.

Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/main.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index 540840e80493..564a82ac3787 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -684,6 +684,9 @@ static int handle_hca_cap_roce(struct mlx5_core_dev *dev, void *set_ctx)
 	       MLX5_ST_SZ_BYTES(roce_cap));
 	MLX5_SET(roce_cap, set_hca_cap, sw_r_roce_src_udp_port, 1);
 
+	if (MLX5_CAP_ROCE_MAX(dev, qp_ooo_transmit_default))
+		MLX5_SET(roce_cap, set_hca_cap, qp_ooo_transmit_default, 1);
+
 	err = set_caps(dev, set_ctx, MLX5_SET_HCA_CAP_OP_MOD_ROCE);
 	return err;
 }
-- 
2.39.2

Re: [PATCH rdma-next 0/3] Enable IB out-of-order by default in mlx5

[Leon Romanovsky]

On Sun, 19 Mar 2023 14:59:29 +0200, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@nvidia.com>
> 
> Hi,
> 
> This series from Or changes default of IB out-of-order feature and
> allows to the RDMA users to decide if they need to wait for completion
> for all segments or it is enough to wait for last segment completion only.
> 
> [...]

Applied, thanks!

[1/3] net/mlx5: Expose bits for enabling out-of-order by default
      https://git.kernel.org/rdma/rdma/c/6e2a3a324aab9d
[2/3] RDMA/mlx5: Disable out-of-order in integrity enabled QPs
      https://git.kernel.org/rdma/rdma/c/742948cc02d523
[3/3] net/mlx5: Set out of order (ooo) by default
      https://git.kernel.org/rdma/rdma/c/f4244e55e4c3a1

Best regards,
-- 
Leon Romanovsky <leon@kernel.org>