[PATCH net 0/3] net/mlx5e: Fix CQE moderation mode reporting and control

[PATCH net 0/3] net/mlx5e: Fix CQE moderation mode reporting and control

[Tariq Toukan]

Hi,

This series by Yael fixes inconsistencies in how mlx5e reports and
configures the CQE moderation mode through ethtool.

mlx5e_params keeps two representations of CQE moderation: the private
flags (MLX5E_PFLAG_RX_CQE_BASED_MODER, MLX5E_PFLAG_TX_CQE_BASED_MODER)
and the fields rx_moder_use_cqe_mode / tx_moder_use_cqe_mode. Ethtool
get/set coalesce uses the fields, while the priv-flags path uses the
pflags. When the two disagree, the reported CQE moderation state is
wrong. In addition, commit bc541621f8ba ("net/mlx5e: Support ethtool cq
mode") added ETHTOOL_COALESCE_USE_CQE to the NIC ethtool ops but not to
the representor ones.

Patch 1 syncs the pflags with the capability-derived fields in
mlx5e_build_nic_params(), so that after channel attach or reopen the
private flags no longer remain zero while the fields reflect the
hardware configuration.

Patch 2 fixes the opposite direction. set_pflag_cqe_based_moder() now
also updates rx_moder_use_cqe_mode / tx_moder_use_cqe_mode, and resets
the RX/TX CQ moderation to the defaults of the new period mode, so that
ethtool -c no longer reports a stale CQE mode or stale rx-usecs after
toggling the priv flag.

Patch 3 adds ETHTOOL_COALESCE_USE_CQE to mlx5e_rep_ethtool_ops, so that
setting cqe-mode-rx/tx on a representor is no longer rejected with
EINVAL.

Regards,
Tariq

Yael Chemla (3):
  net/mlx5e: Sync CQE moderation pflags with coalesce params
  net/mlx5e: Sync CQE moderation params with pflags in
    set_pflag_cqe_based_moder
  net/mlx5e: Expose CQE moderation mode on representors

 drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c | 8 ++++++++
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c    | 4 ++++
 drivers/net/ethernet/mellanox/mlx5/core/en_rep.c     | 3 ++-
 3 files changed, 14 insertions(+), 1 deletion(-)


base-commit: af32da41b0327b9c6a37856ba82b6760d6c8d10e
-- 
2.44.0

[PATCH net 1/3] net/mlx5e: Sync CQE moderation pflags with coalesce params

[Tariq Toukan]

From: Yael Chemla <ychemla@nvidia.com>

mlx5e_params keeps two representations of CQE moderation: the private
flags (MLX5E_PFLAG_RX_CQE_BASED_MODER, MLX5E_PFLAG_TX_CQE_BASED_MODER)
and the fields rx_moder_use_cqe_mode / tx_moder_use_cqe_mode, which are
used by ethtool get/set coalesce. Both are used by the driver (ethtool
get/set coalesce, channel open, moderation apply). When they disagree,
CQE moderation reported value is wrong.

Such an issue can be seen on channel attach or reopen:
mlx5e_build_nic_params() runs and sets rx_moder_use_cqe_mode and
tx_moder_use_cqe_mode from device capability but did not set the
corresponding pflags. So after attach/reopen, pflags could remain zero
while the fields reflected hardware configuration.

Fix by syncing pflags in build_nic_params from the capability-derived
fields.

Reproducer:

    ethtool --show-priv-flags eth2 | grep rx_cqe_moder
      rx_cqe_moder       : off
    ethtool --coalesce eth2 adaptive-tx off
    ethtool --show-priv-flags eth2 | grep rx_cqe_moder
      rx_cqe_moder       : on

Fixes: 445a25f6e1a2 ("net/mlx5e: Support updating coalescing configuration without resetting channels")
Signed-off-by: Yael Chemla <ychemla@nvidia.com>
Reviewed-by: Dragos Tatulea <dtatulea@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 4c6060d54fbe..9fcea0395137 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -5407,6 +5407,10 @@ void mlx5e_build_nic_params(struct mlx5e_priv *priv, struct mlx5e_xsk *xsk, u16
 				 MLX5_CAP_GEN(mdev, cq_period_mode_modify);
 	params->rx_moder_use_cqe_mode = !!MLX5_CAP_GEN(mdev, cq_period_start_from_cqe);
 	params->tx_moder_use_cqe_mode = false;
+	MLX5E_SET_PFLAG(params, MLX5E_PFLAG_RX_CQE_BASED_MODER,
+			params->rx_moder_use_cqe_mode);
+	MLX5E_SET_PFLAG(params, MLX5E_PFLAG_TX_CQE_BASED_MODER,
+			params->tx_moder_use_cqe_mode);
 	mlx5e_reset_rx_moderation(&params->rx_cq_moderation, params->rx_moder_use_cqe_mode,
 				  params->rx_dim_enabled);
 	mlx5e_reset_tx_moderation(&params->tx_cq_moderation, params->tx_moder_use_cqe_mode,
-- 
2.44.0

[PATCH net 2/3] net/mlx5e: Sync CQE moderation params with pflags in set_pflag_cqe_based_moder

[Tariq Toukan]

From: Yael Chemla <ychemla@nvidia.com>

When CQE-based moderation is toggled via the priv-flag path
(ethtool --set-priv-flags ... rx_cqe_moder / tx_cqe_moder),
set_pflag_cqe_based_moder() updates only params->pflags and does not
update rx_moder_use_cqe_mode / tx_moder_use_cqe_mode. After the toggle,
--show-priv-flags reflects the new state but get_coalesce (and thus
ethtool -c) still reports the old state because it reads from the
params fields.

Fix by setting new_params.rx_moder_use_cqe_mode or
new_params.tx_moder_use_cqe_mode in set_pflag_cqe_based_moder() when
toggling the corresponding priv flag, so both representations stay
in sync.

Additionally, new_params.rx_cq_moderation / new_params.tx_cq_moderation
are left as a copy of the old params, so ethtool -c reports stale
rx_coalesce_usecs after the toggle. Fix by calling
mlx5e_reset_rx/tx_moderation() on new_params to overwrite usec with
the default for the new period mode before storing it into
priv->channels.params.

Reproducer:

  ethtool -C eth2 adaptive-rx off
  ethtool --show-priv-flags eth2 | grep -E 'rx_cqe_moder|tx_cqe_moder'
    rx_cqe_moder       : on
    tx_cqe_moder       : off
  ethtool -c eth2 | grep -E 'rx-usecs:|CQE mode'
    rx-usecs: 3
    CQE mode RX: on  TX: off
  ethtool --set-priv-flags eth2 rx_cqe_moder off
  ethtool --set-priv-flags eth2 tx_cqe_moder on
  ethtool --show-priv-flags eth2 | grep -E 'rx_cqe_moder|tx_cqe_moder'
    rx_cqe_moder       : off
    tx_cqe_moder       : on
  ethtool -c eth2 | grep -E 'rx-usecs:|CQE mode'
    rx-usecs: 3              # stale: EQE default is 16
    CQE mode RX: on  TX: off # stale: priv-flags changed, CQE mode did not

Fixes: 445a25f6e1a2 ("net/mlx5e: Support updating coalescing configuration without resetting channels")
Signed-off-by: Yael Chemla <ychemla@nvidia.com>
Reviewed-by: Dragos Tatulea <dtatulea@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
index 3ed59ced0407..c2bee69c26af 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
@@ -2193,12 +2193,20 @@ static int set_pflag_cqe_based_moder(struct net_device *netdev, bool enable,
 		mlx5e_channels_rx_toggle_dim(&priv->channels);
 		MLX5E_SET_PFLAG(&new_params, MLX5E_PFLAG_RX_CQE_BASED_MODER,
 				cq_period_mode);
+		new_params.rx_moder_use_cqe_mode = enable;
+		mlx5e_reset_rx_moderation(&new_params.rx_cq_moderation,
+					  cq_period_mode,
+					  new_params.rx_dim_enabled);
 	} else {
 		mlx5e_reset_tx_channels_moderation(&priv->channels, cq_period_mode,
 						   false, true);
 		mlx5e_channels_tx_toggle_dim(&priv->channels);
 		MLX5E_SET_PFLAG(&new_params, MLX5E_PFLAG_TX_CQE_BASED_MODER,
 				cq_period_mode);
+		new_params.tx_moder_use_cqe_mode = enable;
+		mlx5e_reset_tx_moderation(&new_params.tx_cq_moderation,
+					  cq_period_mode,
+					  new_params.tx_dim_enabled);
 	}
 
 	/* Update pflags of existing channels without resetting them */
-- 
2.44.0

[PATCH net 3/3] net/mlx5e: Expose CQE moderation mode on representors

[Tariq Toukan]

From: Yael Chemla <ychemla@nvidia.com>

commit bc541621f8ba ("net/mlx5e: Support ethtool cq mode") added
ETHTOOL_COALESCE_USE_CQE to the NIC ethtool ops but omitted it from
mlx5e_rep_ethtool_ops.supported_coalesce_params. The ethtool netlink
layer rejects any attempt to set cqe-mode-rx/tx on a representor
with EINVAL.

Add ETHTOOL_COALESCE_USE_CQE to mlx5e_rep_ethtool_ops.
mlx5e_ethtool_set_coalesce() already handles it correctly.

Fixes: bc541621f8ba ("net/mlx5e: Support ethtool cq mode")
Signed-off-by: Yael Chemla <ychemla@nvidia.com>
Reviewed-by: Dragos Tatulea <dtatulea@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en_rep.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
index 88a170e40bd9..4fb132a57bca 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
@@ -416,7 +416,8 @@ static u32 mlx5e_rep_get_rxfh_indir_size(struct net_device *netdev)
 static const struct ethtool_ops mlx5e_rep_ethtool_ops = {
 	.supported_coalesce_params = ETHTOOL_COALESCE_USECS |
 				     ETHTOOL_COALESCE_MAX_FRAMES |
-				     ETHTOOL_COALESCE_USE_ADAPTIVE,
+				     ETHTOOL_COALESCE_USE_ADAPTIVE |
+				     ETHTOOL_COALESCE_USE_CQE,
 	.op_needs_rtnl	   = ETHTOOL_OP_NEEDS_RTNL_SCHANNELS |
 			     ETHTOOL_OP_NEEDS_RTNL_SRINGPARAM |
 			     ETHTOOL_OP_NEEDS_RTNL_GLINK,
-- 
2.44.0

Re: [PATCH net 0/3] net/mlx5e: Fix CQE moderation mode reporting and control

[netdev-bot+sinfo]

Hi!

This is an automated message. This series looks like a fix, but its
commit messages seem to be missing some information:

 - What hardware the change was tested on. For driver fixes please
   mention the device (and if relevant firmware version) used for
   testing, or say that the change was not tested on real hardware.

Please do not repost the series just to address the above. Instead,
reply to this email with the missing information, so that reviewers
can take it into account. If the series needs another revision for
other reasons, please include the information in the commit messages
then.

The evaluation is done by an LLM so it may be wrong, if you think
that is the case please reply and explain.