[PATCH net-next 0/7] net/mlx5e: Reduce interface downtime on configuration change

[PATCH net-next 0/7] net/mlx5e: Reduce interface downtime on configuration change

[Tariq Toukan]

Hi,

This series significantly reduces the interface downtime while swapping
channels during a configuration change, on capable devices.

Here we remove an old requirement on operations ordering that became
obsolete on recent capable devices. This helps cutting the downtime by a
factor of magnitude, ~80% in our example.

Perf numbers:
Measured the number of dropped packets in a simple ping flood test,
during a configuration change operation, that switches the number of
channels from 247 to 248.

Before: 71 packets lost
After:  15 packets lost, ~80% saving.

Regards,
Tariq

Tariq Toukan (7):
  net/mlx5e: Enhance function structures for self loopback prevention
    application
  net/mlx5e: Use TIR API in mlx5e_modify_tirs_lb()
  net/mlx5e: Allow setting self loopback prevention bits on TIR init
  net/mlx5: IPoIB, set self loopback prevention in TIR init
  net/mlx5e: Do not re-apply TIR loopback configuration if not necessary
  net/mlx5e: Pass old channels as argument to mlx5e_switch_priv_channels
  net/mlx5e: Defer channels closure to reduce interface down time

 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  4 +-
 .../net/ethernet/mellanox/mlx5/core/en/rss.c  |  2 +
 .../net/ethernet/mellanox/mlx5/core/en/rss.h  |  1 +
 .../ethernet/mellanox/mlx5/core/en/rx_res.c   |  7 +++
 .../ethernet/mellanox/mlx5/core/en/rx_res.h   |  1 +
 .../net/ethernet/mellanox/mlx5/core/en/tir.c  | 29 ++++++++++--
 .../net/ethernet/mellanox/mlx5/core/en/tir.h  |  3 ++
 .../ethernet/mellanox/mlx5/core/en_common.c   | 46 +++++++++----------
 .../net/ethernet/mellanox/mlx5/core/en_main.c | 31 +++++++++----
 .../ethernet/mellanox/mlx5/core/en_selftest.c |  4 +-
 .../ethernet/mellanox/mlx5/core/ipoib/ipoib.c |  7 ++-
 11 files changed, 94 insertions(+), 41 deletions(-)


base-commit: d550d63d0082268a31e93a10c64cbc2476b98b24
-- 
2.31.1

[PATCH net-next 1/7] net/mlx5e: Enhance function structures for self loopback prevention application

[Tariq Toukan]

The re-application of self loopback prevention attributes in TIRs is
necessary in old firmwares (where tis_tir_td_order cap is cleared) after
recreation of SQs.

However, this is not needed in new firmware with tis_tir_td_order=1.

As a preparation patch, enhance the function structures to differentiate
between an explicit loopback prevention configuration apply, and the
re-apply operation required by old firmware.

Loopback selftests should now call mlx5e_modify_tirs_lb() directly, as
their use case is not related to the firmware limitation.

Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
Reviewed-by: Carolina Jubran <cjubran@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en.h     |  4 +++-
 .../net/ethernet/mellanox/mlx5/core/en_common.c  | 16 ++++++++++++----
 .../net/ethernet/mellanox/mlx5/core/en_main.c    |  2 +-
 .../ethernet/mellanox/mlx5/core/en_selftest.c    |  4 ++--
 .../ethernet/mellanox/mlx5/core/ipoib/ipoib.c    |  2 +-
 5 files changed, 19 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 14e3207b14e7..a25588fe2773 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -1154,7 +1154,9 @@ extern const struct ethtool_ops mlx5e_ethtool_ops;
 int mlx5e_create_mkey(struct mlx5_core_dev *mdev, u32 pdn, u32 *mkey);
 int mlx5e_create_mdev_resources(struct mlx5_core_dev *mdev, bool create_tises);
 void mlx5e_destroy_mdev_resources(struct mlx5_core_dev *mdev);
-int mlx5e_refresh_tirs(struct mlx5e_priv *priv, bool enable_uc_lb,
+int mlx5e_modify_tirs_lb(struct mlx5_core_dev *mdev, bool enable_uc_lb,
+			 bool enable_mc_lb);
+int mlx5e_refresh_tirs(struct mlx5_core_dev *mdev, bool enable_uc_lb,
 		       bool enable_mc_lb);
 void mlx5e_mkey_set_relaxed_ordering(struct mlx5_core_dev *mdev, void *mkc);
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
index 30424ccad584..376a018b2db1 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
@@ -247,10 +247,9 @@ void mlx5e_destroy_mdev_resources(struct mlx5_core_dev *mdev)
 	memset(res, 0, sizeof(*res));
 }
 
-int mlx5e_refresh_tirs(struct mlx5e_priv *priv, bool enable_uc_lb,
-		       bool enable_mc_lb)
+int mlx5e_modify_tirs_lb(struct mlx5_core_dev *mdev, bool enable_uc_lb,
+			 bool enable_mc_lb)
 {
-	struct mlx5_core_dev *mdev = priv->mdev;
 	struct mlx5e_tir *tir;
 	u8 lb_flags = 0;
 	int err  = 0;
@@ -285,7 +284,16 @@ int mlx5e_refresh_tirs(struct mlx5e_priv *priv, bool enable_uc_lb,
 
 	kvfree(in);
 	if (err)
-		netdev_err(priv->netdev, "refresh tir(0x%x) failed, %d\n", tirn, err);
+		mlx5_core_err(mdev,
+			      "modify tir(0x%x) enable_lb uc(%d) mc(%d) failed, %d\n",
+			      tirn,
+			      enable_uc_lb, enable_mc_lb, err);
 
 	return err;
 }
+
+int mlx5e_refresh_tirs(struct mlx5_core_dev *mdev, bool enable_uc_lb,
+		       bool enable_mc_lb)
+{
+	return mlx5e_modify_tirs_lb(mdev, enable_uc_lb, enable_mc_lb);
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index a56825921c23..8e13f2542d8d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -6130,7 +6130,7 @@ static void mlx5e_nic_disable(struct mlx5e_priv *priv)
 
 static int mlx5e_update_nic_rx(struct mlx5e_priv *priv)
 {
-	return mlx5e_refresh_tirs(priv, false, false);
+	return mlx5e_refresh_tirs(priv->mdev, false, false);
 }
 
 static const struct mlx5e_profile mlx5e_nic_profile = {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c b/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c
index 2f7a543feca6..fcad464bc4d5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c
@@ -214,7 +214,7 @@ static int mlx5e_test_loopback_setup(struct mlx5e_priv *priv,
 			return err;
 	}
 
-	err = mlx5e_refresh_tirs(priv, true, false);
+	err = mlx5e_modify_tirs_lb(priv->mdev, true, false);
 	if (err)
 		goto out;
 
@@ -243,7 +243,7 @@ static void mlx5e_test_loopback_cleanup(struct mlx5e_priv *priv,
 		mlx5_nic_vport_update_local_lb(priv->mdev, false);
 
 	dev_remove_pack(&lbtp->pt);
-	mlx5e_refresh_tirs(priv, false, false);
+	mlx5e_modify_tirs_lb(priv->mdev, false, false);
 }
 
 static int mlx5e_cond_loopback(struct mlx5e_priv *priv)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
index 79ae3a51a4b3..49ab0de762c9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
@@ -316,7 +316,7 @@ void mlx5i_destroy_underlay_qp(struct mlx5_core_dev *mdev, u32 qpn)
 
 int mlx5i_update_nic_rx(struct mlx5e_priv *priv)
 {
-	return mlx5e_refresh_tirs(priv, true, true);
+	return mlx5e_refresh_tirs(priv->mdev, true, true);
 }
 
 int mlx5i_create_tis(struct mlx5_core_dev *mdev, u32 underlay_qpn, u32 *tisn)
-- 
2.31.1

[PATCH net-next 2/7] net/mlx5e: Use TIR API in mlx5e_modify_tirs_lb()

[Tariq Toukan]

Extend the TIR API and use it in mlx5e_modify_tirs_lb() instead of the
explicit modify_tir code.

Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
Reviewed-by: Carolina Jubran <cjubran@nvidia.com>
Reviewed-by: Dragos Tatulea <dtatulea@nvidia.com>
---
 .../net/ethernet/mellanox/mlx5/core/en/tir.c  | 29 +++++++++++++++++--
 .../net/ethernet/mellanox/mlx5/core/en/tir.h  |  3 ++
 .../ethernet/mellanox/mlx5/core/en_common.c   | 29 +++++--------------
 3 files changed, 37 insertions(+), 24 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c
index 19499072f67f..0b55e77f19c8 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c
@@ -146,6 +146,31 @@ void mlx5e_tir_builder_build_direct(struct mlx5e_tir_builder *builder)
 	MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FN_INVERTED_XOR8);
 }
 
+static void mlx5e_tir_context_self_lb_block(void *tirc, bool enable_uc_lb,
+					    bool enable_mc_lb)
+{
+	u8 lb_flags = 0;
+
+	if (enable_uc_lb)
+		lb_flags = MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST;
+	if (enable_mc_lb)
+		lb_flags |= MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST;
+
+	MLX5_SET(tirc, tirc, self_lb_block, lb_flags);
+}
+
+void mlx5e_tir_builder_build_self_lb_block(struct mlx5e_tir_builder *builder,
+					   bool enable_uc_lb,
+					   bool enable_mc_lb)
+{
+	void *tirc = mlx5e_tir_builder_get_tirc(builder);
+
+	if (builder->modify)
+		MLX5_SET(modify_tir_in, builder->in, bitmask.self_lb_en, 1);
+
+	mlx5e_tir_context_self_lb_block(tirc, enable_uc_lb, enable_mc_lb);
+}
+
 void mlx5e_tir_builder_build_tls(struct mlx5e_tir_builder *builder)
 {
 	void *tirc = mlx5e_tir_builder_get_tirc(builder);
@@ -153,9 +178,7 @@ void mlx5e_tir_builder_build_tls(struct mlx5e_tir_builder *builder)
 	WARN_ON(builder->modify);
 
 	MLX5_SET(tirc, tirc, tls_en, 1);
-	MLX5_SET(tirc, tirc, self_lb_block,
-		 MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST |
-		 MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST);
+	mlx5e_tir_context_self_lb_block(tirc, true, true);
 }
 
 int mlx5e_tir_init(struct mlx5e_tir *tir, struct mlx5e_tir_builder *builder,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tir.h b/drivers/net/ethernet/mellanox/mlx5/core/en/tir.h
index e8df3aaf6562..958eeb959a19 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/tir.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tir.h
@@ -35,6 +35,9 @@ void mlx5e_tir_builder_build_rss(struct mlx5e_tir_builder *builder,
 				 const struct mlx5e_rss_params_traffic_type *rss_tt,
 				 bool inner);
 void mlx5e_tir_builder_build_direct(struct mlx5e_tir_builder *builder);
+void mlx5e_tir_builder_build_self_lb_block(struct mlx5e_tir_builder *builder,
+					   bool enable_uc_lb,
+					   bool enable_mc_lb);
 void mlx5e_tir_builder_build_tls(struct mlx5e_tir_builder *builder);
 
 struct mlx5_core_dev;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
index 376a018b2db1..fad6b761f622 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
@@ -250,43 +250,30 @@ void mlx5e_destroy_mdev_resources(struct mlx5_core_dev *mdev)
 int mlx5e_modify_tirs_lb(struct mlx5_core_dev *mdev, bool enable_uc_lb,
 			 bool enable_mc_lb)
 {
+	struct mlx5e_tir_builder *builder;
 	struct mlx5e_tir *tir;
-	u8 lb_flags = 0;
 	int err  = 0;
-	u32 tirn = 0;
-	int inlen;
-	void *in;
 
-	inlen = MLX5_ST_SZ_BYTES(modify_tir_in);
-	in = kvzalloc(inlen, GFP_KERNEL);
-	if (!in)
+	builder = mlx5e_tir_builder_alloc(true);
+	if (!builder)
 		return -ENOMEM;
 
-	if (enable_uc_lb)
-		lb_flags = MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST;
-
-	if (enable_mc_lb)
-		lb_flags |= MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST;
-
-	if (lb_flags)
-		MLX5_SET(modify_tir_in, in, ctx.self_lb_block, lb_flags);
-
-	MLX5_SET(modify_tir_in, in, bitmask.self_lb_en, 1);
+	mlx5e_tir_builder_build_self_lb_block(builder, enable_uc_lb,
+					      enable_mc_lb);
 
 	mutex_lock(&mdev->mlx5e_res.hw_objs.td.list_lock);
 	list_for_each_entry(tir, &mdev->mlx5e_res.hw_objs.td.tirs_list, list) {
-		tirn = tir->tirn;
-		err = mlx5_core_modify_tir(mdev, tirn, in);
+		err = mlx5e_tir_modify(tir, builder);
 		if (err)
 			break;
 	}
 	mutex_unlock(&mdev->mlx5e_res.hw_objs.td.list_lock);
 
-	kvfree(in);
+	mlx5e_tir_builder_free(builder);
 	if (err)
 		mlx5_core_err(mdev,
 			      "modify tir(0x%x) enable_lb uc(%d) mc(%d) failed, %d\n",
-			      tirn,
+			      mlx5e_tir_get_tirn(tir),
 			      enable_uc_lb, enable_mc_lb, err);
 
 	return err;
-- 
2.31.1

[PATCH net-next 3/7] net/mlx5e: Allow setting self loopback prevention bits on TIR init

[Tariq Toukan]

Until now, IPoIB was creating TIRs without setting self loopback
prevention, then modifying them in activation stage.

This is a preparation patch, that will be used by IPoIB to init TIRs
properly without the need for following calls of modify_tir.

Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
Reviewed-by: Carolina Jubran <cjubran@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en/rss.c    | 2 ++
 drivers/net/ethernet/mellanox/mlx5/core/en/rss.h    | 1 +
 drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c | 7 +++++++
 drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h | 1 +
 4 files changed, 11 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rss.c b/drivers/net/ethernet/mellanox/mlx5/core/en/rss.c
index c96cbc4b0dbf..88b0e1050d1a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/rss.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rss.c
@@ -231,6 +231,8 @@ mlx5e_rss_create_tir(struct mlx5e_rss *rss, enum mlx5_traffic_types tt,
 				    rqtn, rss_inner);
 	mlx5e_tir_builder_build_packet_merge(builder, pkt_merge_param);
 	rss_tt = mlx5e_rss_get_tt_config(rss, tt);
+	mlx5e_tir_builder_build_self_lb_block(builder, rss->params.self_lb_blk,
+					      rss->params.self_lb_blk);
 	mlx5e_tir_builder_build_rss(builder, &rss->hash, &rss_tt, inner);
 
 	err = mlx5e_tir_init(tir, builder, rss->mdev, true);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rss.h b/drivers/net/ethernet/mellanox/mlx5/core/en/rss.h
index 5fb03cd0a411..17664757a561 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/rss.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rss.h
@@ -23,6 +23,7 @@ struct mlx5e_rss_init_params {
 struct mlx5e_rss_params {
 	bool inner_ft_support;
 	u32 drop_rqn;
+	bool self_lb_blk;
 };
 
 struct mlx5e_rss_params_traffic_type
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c
index ac26a32845d0..55c117b7d8c4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c
@@ -71,6 +71,8 @@ static int mlx5e_rx_res_rss_init_def(struct mlx5e_rx_res *res,
 	rss_params = (struct mlx5e_rss_params) {
 		.inner_ft_support = inner_ft_support,
 		.drop_rqn = res->drop_rqn,
+		.self_lb_blk =
+			res->features & MLX5E_RX_RES_FEATURE_SELF_LB_BLOCK,
 	};
 
 	rss = mlx5e_rss_init(res->mdev, &rss_params, &init_params);
@@ -104,6 +106,8 @@ int mlx5e_rx_res_rss_init(struct mlx5e_rx_res *res, u32 rss_idx, unsigned int in
 	rss_params = (struct mlx5e_rss_params) {
 		.inner_ft_support = inner_ft_support,
 		.drop_rqn = res->drop_rqn,
+		.self_lb_blk =
+			res->features & MLX5E_RX_RES_FEATURE_SELF_LB_BLOCK,
 	};
 
 	rss = mlx5e_rss_init(res->mdev, &rss_params, &init_params);
@@ -346,6 +350,7 @@ static struct mlx5e_rx_res *mlx5e_rx_res_alloc(struct mlx5_core_dev *mdev, unsig
 static int mlx5e_rx_res_channels_init(struct mlx5e_rx_res *res)
 {
 	bool inner_ft_support = res->features & MLX5E_RX_RES_FEATURE_INNER_FT;
+	bool self_lb_blk = res->features & MLX5E_RX_RES_FEATURE_SELF_LB_BLOCK;
 	struct mlx5e_tir_builder *builder;
 	int err = 0;
 	int ix;
@@ -376,6 +381,8 @@ static int mlx5e_rx_res_channels_init(struct mlx5e_rx_res *res)
 					    mlx5e_rqt_get_rqtn(&res->channels[ix].direct_rqt),
 					    inner_ft_support);
 		mlx5e_tir_builder_build_packet_merge(builder, &res->pkt_merge_param);
+		mlx5e_tir_builder_build_self_lb_block(builder, self_lb_blk,
+						      self_lb_blk);
 		mlx5e_tir_builder_build_direct(builder);
 
 		err = mlx5e_tir_init(&res->channels[ix].direct_tir, builder, res->mdev, true);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h
index 65a857c215e1..675780120a20 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h
@@ -21,6 +21,7 @@ enum mlx5e_rx_res_features {
 	MLX5E_RX_RES_FEATURE_INNER_FT = BIT(0),
 	MLX5E_RX_RES_FEATURE_PTP = BIT(1),
 	MLX5E_RX_RES_FEATURE_MULTI_VHCA = BIT(2),
+	MLX5E_RX_RES_FEATURE_SELF_LB_BLOCK = BIT(3),
 };
 
 /* Setup */
-- 
2.31.1

[PATCH net-next 4/7] net/mlx5: IPoIB, set self loopback prevention in TIR init

[Tariq Toukan]

In IPoIB, the self loopback prevention configuration apply in activation
stage has two roles: fulfill a firmware requirement for old firmware
(tis_tir_td_order=0), and update the proper configuration as it was not
set in init.

Here we set the proper configuration in init, to allow skipping the
modify_tirs commands on new firmware in a downstream patch.

Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
Reviewed-by: Carolina Jubran <cjubran@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
index 49ab0de762c9..7f3f6d7edb38 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
@@ -409,6 +409,7 @@ static void mlx5i_destroy_flow_steering(struct mlx5e_priv *priv)
 static int mlx5i_init_rx(struct mlx5e_priv *priv)
 {
 	struct mlx5_core_dev *mdev = priv->mdev;
+	enum mlx5e_rx_res_features features;
 	int err;
 
 	priv->fs = mlx5e_fs_init(priv->profile, mdev,
@@ -427,7 +428,9 @@ static int mlx5i_init_rx(struct mlx5e_priv *priv)
 		goto err_destroy_q_counters;
 	}
 
-	priv->rx_res = mlx5e_rx_res_create(priv->mdev, 0, priv->max_nch, priv->drop_rq.rqn,
+	features = MLX5E_RX_RES_FEATURE_SELF_LB_BLOCK;
+	priv->rx_res = mlx5e_rx_res_create(priv->mdev, features, priv->max_nch,
+					   priv->drop_rq.rqn,
 					   &priv->channels.params.packet_merge,
 					   priv->channels.params.num_channels);
 	if (IS_ERR(priv->rx_res)) {
-- 
2.31.1

[PATCH net-next 5/7] net/mlx5e: Do not re-apply TIR loopback configuration if not necessary

[Tariq Toukan]

On old firmware, (tis_tir_td_order=0), TIR of a transport domain should
either be created after all SQs of the same domain, or TIR.self_lb_en
should be reapplied using MODIFY_TIR, for self loopback filtering to
function correctly.

This is not necessary anymnore on new FW (tis_tir_td_order=1), thus
there's no need for calling modify_tir operations after creating a new
set of SQs to maintain the self loopback prevention functional.

Skip these operations.

This saves O(max_num_channels) MODIFY_TIR firmware commands in
operations like interface up or channels configuration change.

Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
Reviewed-by: Carolina Jubran <cjubran@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en_common.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
index fad6b761f622..1d663c9597a3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
@@ -282,5 +282,8 @@ int mlx5e_modify_tirs_lb(struct mlx5_core_dev *mdev, bool enable_uc_lb,
 int mlx5e_refresh_tirs(struct mlx5_core_dev *mdev, bool enable_uc_lb,
 		       bool enable_mc_lb)
 {
+	if (MLX5_CAP_GEN(mdev, tis_tir_td_order))
+		return 0; /* refresh not needed */
+
 	return mlx5e_modify_tirs_lb(mdev, enable_uc_lb, enable_mc_lb);
 }
-- 
2.31.1

[PATCH net-next 6/7] net/mlx5e: Pass old channels as argument to mlx5e_switch_priv_channels

[Tariq Toukan]

Let the caller function mlx5e_safe_switch_params() maintain a copy
of the old channels, and pass it to mlx5e_switch_priv_channels().

This is in preparation for the next patch.

Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
Reviewed-by: Carolina Jubran <cjubran@nvidia.com>
---
 .../net/ethernet/mellanox/mlx5/core/en_main.c | 25 +++++++++++++------
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 8e13f2542d8d..cd2842dcffcd 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -3356,12 +3356,12 @@ static int mlx5e_switch_priv_params(struct mlx5e_priv *priv,
 }
 
 static int mlx5e_switch_priv_channels(struct mlx5e_priv *priv,
+				      struct mlx5e_channels *old_chs,
 				      struct mlx5e_channels *new_chs,
 				      mlx5e_fp_preactivate preactivate,
 				      void *context)
 {
 	struct net_device *netdev = priv->netdev;
-	struct mlx5e_channels old_chs;
 	int carrier_ok;
 	int err = 0;
 
@@ -3370,7 +3370,6 @@ static int mlx5e_switch_priv_channels(struct mlx5e_priv *priv,
 
 	mlx5e_deactivate_priv_channels(priv);
 
-	old_chs = priv->channels;
 	priv->channels = *new_chs;
 
 	/* New channels are ready to roll, call the preactivate hook if needed
@@ -3379,12 +3378,12 @@ static int mlx5e_switch_priv_channels(struct mlx5e_priv *priv,
 	if (preactivate) {
 		err = preactivate(priv, context);
 		if (err) {
-			priv->channels = old_chs;
+			priv->channels = *old_chs;
 			goto out;
 		}
 	}
 
-	mlx5e_close_channels(&old_chs);
+	mlx5e_close_channels(old_chs);
 	priv->profile->update_rx(priv);
 
 	mlx5e_selq_apply(&priv->selq);
@@ -3403,16 +3402,20 @@ int mlx5e_safe_switch_params(struct mlx5e_priv *priv,
 			     mlx5e_fp_preactivate preactivate,
 			     void *context, bool reset)
 {
-	struct mlx5e_channels *new_chs;
+	struct mlx5e_channels *old_chs, *new_chs;
 	int err;
 
 	reset &= test_bit(MLX5E_STATE_OPENED, &priv->state);
 	if (!reset)
 		return mlx5e_switch_priv_params(priv, params, preactivate, context);
 
+	old_chs = kzalloc(sizeof(*old_chs), GFP_KERNEL);
 	new_chs = kzalloc(sizeof(*new_chs), GFP_KERNEL);
-	if (!new_chs)
-		return -ENOMEM;
+	if (!old_chs || !new_chs) {
+		err = -ENOMEM;
+		goto err_free_chs;
+	}
+
 	new_chs->params = *params;
 
 	mlx5e_selq_prepare_params(&priv->selq, &new_chs->params);
@@ -3421,11 +3424,15 @@ int mlx5e_safe_switch_params(struct mlx5e_priv *priv,
 	if (err)
 		goto err_cancel_selq;
 
-	err = mlx5e_switch_priv_channels(priv, new_chs, preactivate, context);
+	*old_chs = priv->channels;
+
+	err = mlx5e_switch_priv_channels(priv, old_chs, new_chs,
+					 preactivate, context);
 	if (err)
 		goto err_close;
 
 	kfree(new_chs);
+	kfree(old_chs);
 	return 0;
 
 err_close:
@@ -3433,7 +3440,9 @@ int mlx5e_safe_switch_params(struct mlx5e_priv *priv,
 
 err_cancel_selq:
 	mlx5e_selq_cancel(&priv->selq);
+err_free_chs:
 	kfree(new_chs);
+	kfree(old_chs);
 	return err;
 }
 
-- 
2.31.1

[PATCH net-next 7/7] net/mlx5e: Defer channels closure to reduce interface down time

[Tariq Toukan]

Cap bit tis_tir_td_order=1 indicates that an old firmware requirement /
limitation no longer exists. When unset, the latency of several firmware
commands significantly increases with the presence of high number of
co-existing channels (both old and new sets). Hence, we used to close
unneeded old channels before invoking those firmware commands.

Today, on capable devices, this is no longer the case. Minimize the
interface down time by deferring the old channels closure, after the
activation of the new ones.

Perf numbers:
Measured the number of dropped packets in a simple ping flood test,
during a configuration change operation, that switches the number of
channels from 247 to 248.

Before: 71 packets lost
After:  15 packets lost, ~80% saving.

Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
Reviewed-by: Carolina Jubran <cjubran@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index cd2842dcffcd..a4c2c78660b0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -3383,7 +3383,8 @@ static int mlx5e_switch_priv_channels(struct mlx5e_priv *priv,
 		}
 	}
 
-	mlx5e_close_channels(old_chs);
+	if (!MLX5_CAP_GEN(priv->mdev, tis_tir_td_order))
+		mlx5e_close_channels(old_chs);
 	priv->profile->update_rx(priv);
 
 	mlx5e_selq_apply(&priv->selq);
@@ -3431,6 +3432,9 @@ int mlx5e_safe_switch_params(struct mlx5e_priv *priv,
 	if (err)
 		goto err_close;
 
+	if (MLX5_CAP_GEN(priv->mdev, tis_tir_td_order))
+		mlx5e_close_channels(old_chs);
+
 	kfree(new_chs);
 	kfree(old_chs);
 	return 0;
-- 
2.31.1

Re: [PATCH net-next 2/7] net/mlx5e: Use TIR API in mlx5e_modify_tirs_lb()

[Simon Horman]

On Thu, Oct 23, 2025 at 09:43:35AM +0300, Tariq Toukan wrote:

...

> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c
> index 19499072f67f..0b55e77f19c8 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c
> @@ -146,6 +146,31 @@ void mlx5e_tir_builder_build_direct(struct mlx5e_tir_builder *builder)
>  	MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FN_INVERTED_XOR8);
>  }
>  
> +static void mlx5e_tir_context_self_lb_block(void *tirc, bool enable_uc_lb,
> +					    bool enable_mc_lb)
> +{
> +	u8 lb_flags = 0;
> +
> +	if (enable_uc_lb)
> +		lb_flags = MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST;
> +	if (enable_mc_lb)
> +		lb_flags |= MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST;
> +
> +	MLX5_SET(tirc, tirc, self_lb_block, lb_flags);
> +}
> +
> +void mlx5e_tir_builder_build_self_lb_block(struct mlx5e_tir_builder *builder,
> +					   bool enable_uc_lb,
> +					   bool enable_mc_lb)
> +{
> +	void *tirc = mlx5e_tir_builder_get_tirc(builder);
> +
> +	if (builder->modify)
> +		MLX5_SET(modify_tir_in, builder->in, bitmask.self_lb_en, 1);
> +
> +	mlx5e_tir_context_self_lb_block(tirc, enable_uc_lb, enable_mc_lb);
> +}
> +

...

> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
> index 376a018b2db1..fad6b761f622 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
> @@ -250,43 +250,30 @@ void mlx5e_destroy_mdev_resources(struct mlx5_core_dev *mdev)
>  int mlx5e_modify_tirs_lb(struct mlx5_core_dev *mdev, bool enable_uc_lb,
>  			 bool enable_mc_lb)

...

> -	if (enable_uc_lb)
> -		lb_flags = MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST;
> -
> -	if (enable_mc_lb)
> -		lb_flags |= MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST;
> -
> -	if (lb_flags)
> -		MLX5_SET(modify_tir_in, in, ctx.self_lb_block, lb_flags);
> -
> -	MLX5_SET(modify_tir_in, in, bitmask.self_lb_en, 1);
> +	mlx5e_tir_builder_build_self_lb_block(builder, enable_uc_lb,
> +					      enable_mc_lb);

Hi,

Maybe I'm reading this wrong, and possibly it is not important,
but it seems to me that the update above reverses the

...
order of the MLX5_SET() invocations.

Re: [PATCH net-next 2/7] net/mlx5e: Use TIR API in mlx5e_modify_tirs_lb()

[Simon Horman]

On Thu, Oct 23, 2025 at 09:43:35AM +0300, Tariq Toukan wrote:

...

> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
> index 376a018b2db1..fad6b761f622 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
> @@ -250,43 +250,30 @@ void mlx5e_destroy_mdev_resources(struct mlx5_core_dev *mdev)
>  int mlx5e_modify_tirs_lb(struct mlx5_core_dev *mdev, bool enable_uc_lb,
>  			 bool enable_mc_lb)

...

>  	list_for_each_entry(tir, &mdev->mlx5e_res.hw_objs.td.tirs_list, list) {
> -		tirn = tir->tirn;
> -		err = mlx5_core_modify_tir(mdev, tirn, in);
> +		err = mlx5e_tir_modify(tir, builder);
>  		if (err)
>  			break;
>  	}
>  	mutex_unlock(&mdev->mlx5e_res.hw_objs.td.list_lock);
>  
> -	kvfree(in);
> +	mlx5e_tir_builder_free(builder);
>  	if (err)
>  		mlx5_core_err(mdev,
>  			      "modify tir(0x%x) enable_lb uc(%d) mc(%d) failed, %d\n",
> -			      tirn,
> +			      mlx5e_tir_get_tirn(tir),

Sorry, for not noticing this before sending my previous email.

Coccinelle complains about the line above like this:

.../en_common.c:276:28-31: ERROR: invalid reference to the index variable of the iterator on line 265

I think this is a false positive because the problem only occurs if
the list iteration runs to completion. But err guards against
tir being used in that case.

But, perhaps, to be on the safe side, it would be good practice
to stash tir somewhere?

>  			      enable_uc_lb, enable_mc_lb, err);
>  
>  	return err;
> -- 
> 2.31.1
> 
> 

Re: [PATCH net-next 2/7] net/mlx5e: Use TIR API in mlx5e_modify_tirs_lb()

[Tariq Toukan]

On 23/10/2025 16:02, Simon Horman wrote:
> On Thu, Oct 23, 2025 at 09:43:35AM +0300, Tariq Toukan wrote:
> 
> ...
> 
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c
>> index 19499072f67f..0b55e77f19c8 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tir.c
>> @@ -146,6 +146,31 @@ void mlx5e_tir_builder_build_direct(struct mlx5e_tir_builder *builder)
>>   	MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FN_INVERTED_XOR8);
>>   }
>>   
>> +static void mlx5e_tir_context_self_lb_block(void *tirc, bool enable_uc_lb,
>> +					    bool enable_mc_lb)
>> +{
>> +	u8 lb_flags = 0;
>> +
>> +	if (enable_uc_lb)
>> +		lb_flags = MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST;
>> +	if (enable_mc_lb)
>> +		lb_flags |= MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST;
>> +
>> +	MLX5_SET(tirc, tirc, self_lb_block, lb_flags);
>> +}
>> +
>> +void mlx5e_tir_builder_build_self_lb_block(struct mlx5e_tir_builder *builder,
>> +					   bool enable_uc_lb,
>> +					   bool enable_mc_lb)
>> +{
>> +	void *tirc = mlx5e_tir_builder_get_tirc(builder);
>> +
>> +	if (builder->modify)
>> +		MLX5_SET(modify_tir_in, builder->in, bitmask.self_lb_en, 1);
>> +
>> +	mlx5e_tir_context_self_lb_block(tirc, enable_uc_lb, enable_mc_lb);
>> +}
>> +
> 
> ...
> 
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
>> index 376a018b2db1..fad6b761f622 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
>> @@ -250,43 +250,30 @@ void mlx5e_destroy_mdev_resources(struct mlx5_core_dev *mdev)
>>   int mlx5e_modify_tirs_lb(struct mlx5_core_dev *mdev, bool enable_uc_lb,
>>   			 bool enable_mc_lb)
> 
> ...
> 
>> -	if (enable_uc_lb)
>> -		lb_flags = MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST;
>> -
>> -	if (enable_mc_lb)
>> -		lb_flags |= MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST;
>> -
>> -	if (lb_flags)
>> -		MLX5_SET(modify_tir_in, in, ctx.self_lb_block, lb_flags);
>> -
>> -	MLX5_SET(modify_tir_in, in, bitmask.self_lb_en, 1);
>> +	mlx5e_tir_builder_build_self_lb_block(builder, enable_uc_lb,
>> +					      enable_mc_lb);
> 
> Hi,
> 
> Maybe I'm reading this wrong, and possibly it is not important,
> but it seems to me that the update above reverses the
> 
> ...
> order of the MLX5_SET() invocations.
> 

The order here is not important.
The FW command is filled in any order and only then FW is called.

Re: [PATCH net-next 2/7] net/mlx5e: Use TIR API in mlx5e_modify_tirs_lb()

[Tariq Toukan]

On 23/10/2025 16:31, Simon Horman wrote:
> On Thu, Oct 23, 2025 at 09:43:35AM +0300, Tariq Toukan wrote:
> 
> ...
> 
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
>> index 376a018b2db1..fad6b761f622 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
>> @@ -250,43 +250,30 @@ void mlx5e_destroy_mdev_resources(struct mlx5_core_dev *mdev)
>>   int mlx5e_modify_tirs_lb(struct mlx5_core_dev *mdev, bool enable_uc_lb,
>>   			 bool enable_mc_lb)
> 
> ...
> 
>>   	list_for_each_entry(tir, &mdev->mlx5e_res.hw_objs.td.tirs_list, list) {
>> -		tirn = tir->tirn;
>> -		err = mlx5_core_modify_tir(mdev, tirn, in);
>> +		err = mlx5e_tir_modify(tir, builder);
>>   		if (err)
>>   			break;
>>   	}
>>   	mutex_unlock(&mdev->mlx5e_res.hw_objs.td.list_lock);
>>   
>> -	kvfree(in);
>> +	mlx5e_tir_builder_free(builder);
>>   	if (err)
>>   		mlx5_core_err(mdev,
>>   			      "modify tir(0x%x) enable_lb uc(%d) mc(%d) failed, %d\n",
>> -			      tirn,
>> +			      mlx5e_tir_get_tirn(tir),
> 
> Sorry, for not noticing this before sending my previous email.
> 
> Coccinelle complains about the line above like this:
> 
> .../en_common.c:276:28-31: ERROR: invalid reference to the index variable of the iterator on line 265
> 
> I think this is a false positive because the problem only occurs if
> the list iteration runs to completion. But err guards against
> tir being used in that case.
> 

Exactly.

> But, perhaps, to be on the safe side, it would be good practice
> to stash tir somewhere?
> 

I tried to keep the error print out of the critical lock section.
It's not time-sensitive so optimization is not really needed.
I can simply move the print inside where it belongs.

>>   			      enable_uc_lb, enable_mc_lb, err);
>>   
>>   	return err;
>> -- 
>> 2.31.1
>>
>>
> 

Re: [PATCH net-next 2/7] net/mlx5e: Use TIR API in mlx5e_modify_tirs_lb()

[Paolo Abeni]

On 10/26/25 1:50 PM, Tariq Toukan wrote:
> On 23/10/2025 16:31, Simon Horman wrote:
>> On Thu, Oct 23, 2025 at 09:43:35AM +0300, Tariq Toukan wrote:
>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
>>> index 376a018b2db1..fad6b761f622 100644
>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
>>> @@ -250,43 +250,30 @@ void mlx5e_destroy_mdev_resources(struct mlx5_core_dev *mdev)
>>>   int mlx5e_modify_tirs_lb(struct mlx5_core_dev *mdev, bool enable_uc_lb,
>>>   			 bool enable_mc_lb)
>>
>> ...
>>
>>>   	list_for_each_entry(tir, &mdev->mlx5e_res.hw_objs.td.tirs_list, list) {
>>> -		tirn = tir->tirn;
>>> -		err = mlx5_core_modify_tir(mdev, tirn, in);
>>> +		err = mlx5e_tir_modify(tir, builder);
>>>   		if (err)
>>>   			break;
>>>   	}
>>>   	mutex_unlock(&mdev->mlx5e_res.hw_objs.td.list_lock);
>>>   
>>> -	kvfree(in);
>>> +	mlx5e_tir_builder_free(builder);
>>>   	if (err)
>>>   		mlx5_core_err(mdev,
>>>   			      "modify tir(0x%x) enable_lb uc(%d) mc(%d) failed, %d\n",
>>> -			      tirn,
>>> +			      mlx5e_tir_get_tirn(tir),
>>
>> Sorry, for not noticing this before sending my previous email.
>>
>> Coccinelle complains about the line above like this:
>>
>> .../en_common.c:276:28-31: ERROR: invalid reference to the index variable of the iterator on line 265
>>
>> I think this is a false positive because the problem only occurs if
>> the list iteration runs to completion. But err guards against
>> tir being used in that case.
>>
> 
> Exactly.
> 
>> But, perhaps, to be on the safe side, it would be good practice
>> to stash tir somewhere?
>>
> 
> I tried to keep the error print out of the critical lock section.
> It's not time-sensitive so optimization is not really needed.
> I can simply move the print inside where it belongs.

Yes please!

Also the printk is performed on error, should not affect the path
critical path performances even if that would be time-sensitive.

/P