[PATCH net] Revert "net/mlx5e: Update and set Xon/Xoff upon MTU set"

[PATCH net] Revert "net/mlx5e: Update and set Xon/Xoff upon MTU set"

[Jakub Kicinski]

This reverts commit ceddedc969f0532b7c62ca971ee50d519d2bc0cb.

Commit in question breaks the mapping of PGs to pools for some SKUs.
Specifically multi-host NICs seem to be shipped with a custom buffer
configuration which maps the lossy PG to pool 4. But the bad commit
overrides this with pool 0 which does not have sufficient buffer space
reserved. Resulting in ~40% packet loss. The commit also breaks BMC /
OOB connection completely (100% packet loss).

Revert, similarly to commit 3fbfe251cc9f ("Revert "net/mlx5e: Update and
set Xon/Xoff upon port speed set""). The breakage is exactly the same,
the only difference is that quoted commit would break the NIC immediately
on boot, and the currently reverted commit only when MTU is changed.

Note: "good" kernels do not restore the configuration, so downgrade isn't
enough to recover machines. A NIC power cycle seems to be necessary to
return to a healthy state (or overriding the relevant registers using
a custom patch).

Fixes: ceddedc969f0 ("net/mlx5e: Update and set Xon/Xoff upon MTU set")
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
CC: saeedm@nvidia.com
CC: leon@kernel.org
CC: tariqt@nvidia.com
CC: mbloch@nvidia.com
CC: alazar@nvidia.com
CC: linux-rdma@vger.kernel.org
---
 .../mellanox/mlx5/core/en/port_buffer.h         | 12 ------------
 .../net/ethernet/mellanox/mlx5/core/en_main.c   | 17 +----------------
 2 files changed, 1 insertion(+), 28 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/port_buffer.h b/drivers/net/ethernet/mellanox/mlx5/core/en/port_buffer.h
index 66d276a1be83..f4a19ffbb641 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/port_buffer.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/port_buffer.h
@@ -66,23 +66,11 @@ struct mlx5e_port_buffer {
 	struct mlx5e_bufferx_reg  buffer[MLX5E_MAX_NETWORK_BUFFER];
 };
 
-#ifdef CONFIG_MLX5_CORE_EN_DCB
 int mlx5e_port_manual_buffer_config(struct mlx5e_priv *priv,
 				    u32 change, unsigned int mtu,
 				    struct ieee_pfc *pfc,
 				    u32 *buffer_size,
 				    u8 *prio2buffer);
-#else
-static inline int
-mlx5e_port_manual_buffer_config(struct mlx5e_priv *priv,
-				u32 change, unsigned int mtu,
-				void *pfc,
-				u32 *buffer_size,
-				u8 *prio2buffer)
-{
-	return 0;
-}
-#endif
 
 int mlx5e_port_query_buffer(struct mlx5e_priv *priv,
 			    struct mlx5e_port_buffer *port_buffer);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 15eded36b872..21bb88c5d3dc 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -49,7 +49,6 @@
 #include "en.h"
 #include "en/dim.h"
 #include "en/txrx.h"
-#include "en/port_buffer.h"
 #include "en_tc.h"
 #include "en_rep.h"
 #include "en_accel/ipsec.h"
@@ -3041,11 +3040,9 @@ int mlx5e_set_dev_port_mtu(struct mlx5e_priv *priv)
 	struct mlx5e_params *params = &priv->channels.params;
 	struct net_device *netdev = priv->netdev;
 	struct mlx5_core_dev *mdev = priv->mdev;
-	u16 mtu, prev_mtu;
+	u16 mtu;
 	int err;
 
-	mlx5e_query_mtu(mdev, params, &prev_mtu);
-
 	err = mlx5e_set_mtu(mdev, params, params->sw_mtu);
 	if (err)
 		return err;
@@ -3055,18 +3052,6 @@ int mlx5e_set_dev_port_mtu(struct mlx5e_priv *priv)
 		netdev_warn(netdev, "%s: VPort MTU %d is different than netdev mtu %d\n",
 			    __func__, mtu, params->sw_mtu);
 
-	if (mtu != prev_mtu && MLX5_BUFFER_SUPPORTED(mdev)) {
-		err = mlx5e_port_manual_buffer_config(priv, 0, mtu,
-						      NULL, NULL, NULL);
-		if (err) {
-			netdev_warn(netdev, "%s: Failed to set Xon/Xoff values with MTU %d (err %d), setting back to previous MTU %d\n",
-				    __func__, mtu, err, prev_mtu);
-
-			mlx5e_set_mtu(mdev, params, prev_mtu);
-			return err;
-		}
-	}
-
 	params->sw_mtu = mtu;
 	return 0;
 }
-- 
2.51.0

Re: [PATCH net] Revert "net/mlx5e: Update and set Xon/Xoff upon MTU set"

[Tariq Toukan]

On 29/09/2025 21:15, Jakub Kicinski wrote:
> This reverts commit ceddedc969f0532b7c62ca971ee50d519d2bc0cb.
> 
> Commit in question breaks the mapping of PGs to pools for some SKUs.
> Specifically multi-host NICs seem to be shipped with a custom buffer
> configuration which maps the lossy PG to pool 4. But the bad commit
> overrides this with pool 0 which does not have sufficient buffer space
> reserved. Resulting in ~40% packet loss. The commit also breaks BMC /
> OOB connection completely (100% packet loss).
> 
> Revert, similarly to commit 3fbfe251cc9f ("Revert "net/mlx5e: Update and
> set Xon/Xoff upon port speed set""). The breakage is exactly the same,
> the only difference is that quoted commit would break the NIC immediately
> on boot, and the currently reverted commit only when MTU is changed.
> 
> Note: "good" kernels do not restore the configuration, so downgrade isn't
> enough to recover machines. A NIC power cycle seems to be necessary to
> return to a healthy state (or overriding the relevant registers using
> a custom patch).
> 
> Fixes: ceddedc969f0 ("net/mlx5e: Update and set Xon/Xoff upon MTU set")
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---

Reviewed-by: Tariq Toukan <tariqt@nvidia.com>

Thanks for your revert.
We'll post fixed versions when ready.

> CC: saeedm@nvidia.com
> CC: leon@kernel.org
> CC: tariqt@nvidia.com
> CC: mbloch@nvidia.com
> CC: alazar@nvidia.com
> CC: linux-rdma@vger.kernel.org
> ---

Re: [PATCH net] Revert "net/mlx5e: Update and set Xon/Xoff upon MTU set"

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net-next.git (main)
by Paolo Abeni <pabeni@redhat.com>:

On Mon, 29 Sep 2025 11:15:29 -0700 you wrote:
> This reverts commit ceddedc969f0532b7c62ca971ee50d519d2bc0cb.
> 
> Commit in question breaks the mapping of PGs to pools for some SKUs.
> Specifically multi-host NICs seem to be shipped with a custom buffer
> configuration which maps the lossy PG to pool 4. But the bad commit
> overrides this with pool 0 which does not have sufficient buffer space
> reserved. Resulting in ~40% packet loss. The commit also breaks BMC /
> OOB connection completely (100% packet loss).
> 
> [...]

Here is the summary with links:
  - [net] Revert "net/mlx5e: Update and set Xon/Xoff upon MTU set"
    https://git.kernel.org/netdev/net-next/c/6f5dacf88a32

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html