[PATCH mlx5-next] RDMA/mlx5: Use sq timestamp as QP timestamp when RoCE is disabled

[PATCH mlx5-next] RDMA/mlx5: Use sq timestamp as QP timestamp when RoCE is disabled

[Leon Romanovsky]

From: Or Har-Toov <ohartoov@nvidia.com>

When creating a QP, one of the attributes is TS format (timestamp).
In some devices, we have a limitation that all QPs should have the same
ts_format. The ts_format is chosen based on the device's capability.
The qp_ts_format cap resides under the RoCE caps table, and the
cap will be 0 when RoCE is disabled. So when RoCE is disabled, the
value that should be queried is sq_ts_format under HCA caps.

Consider the case when the system supports REAL_TIME_TS format (0x2),
some QPs are created with REAL_TIME_TS as ts_format, and afterwards
RoCE gets disabled. When trying to construct a new QP, we can't use
the qp_ts_format, that is queried from the RoCE caps table, Since it
leads to passing 0x0 (FREE_RUNNING_TS) as the value of the qp_ts_format,
which is different than the ts_format of the previously allocated
QPs REAL_TIME_TS format (0x2).

Thus, to resolve this, read the sq_ts_format, which also reflect
the supported ts format for the QP when RoCE is disabled.

Fixes: 4806f1e2fee8 ("net/mlx5: Set QP timestamp mode to default")
Signed-off-by: Maher Sanalla <msanalla@nvidia.com>
Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 include/linux/mlx5/qp.h | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/include/linux/mlx5/qp.h b/include/linux/mlx5/qp.h
index f0e55bf3ec8b..ad1ce650146c 100644
--- a/include/linux/mlx5/qp.h
+++ b/include/linux/mlx5/qp.h
@@ -576,9 +576,12 @@ static inline const char *mlx5_qp_state_str(int state)
 
 static inline int mlx5_get_qp_default_ts(struct mlx5_core_dev *dev)
 {
-	return !MLX5_CAP_ROCE(dev, qp_ts_format) ?
-		       MLX5_TIMESTAMP_FORMAT_FREE_RUNNING :
-		       MLX5_TIMESTAMP_FORMAT_DEFAULT;
+	u8 supported_ts_cap = mlx5_get_roce_state(dev) ?
+			      MLX5_CAP_ROCE(dev, qp_ts_format) :
+			      MLX5_CAP_GEN(dev, sq_ts_format);
+
+	return supported_ts_cap ? MLX5_TIMESTAMP_FORMAT_DEFAULT :
+	       MLX5_TIMESTAMP_FORMAT_FREE_RUNNING;
 }
 
 #endif /* MLX5_QP_H */
-- 
2.45.2

Re: [PATCH mlx5-next] RDMA/mlx5: Use sq timestamp as QP timestamp when RoCE is disabled

[Simon Horman]

On Sun, Jun 16, 2024 at 07:10:36PM +0300, Leon Romanovsky wrote:
> From: Or Har-Toov <ohartoov@nvidia.com>
> 
> When creating a QP, one of the attributes is TS format (timestamp).
> In some devices, we have a limitation that all QPs should have the same
> ts_format. The ts_format is chosen based on the device's capability.
> The qp_ts_format cap resides under the RoCE caps table, and the
> cap will be 0 when RoCE is disabled. So when RoCE is disabled, the
> value that should be queried is sq_ts_format under HCA caps.
> 
> Consider the case when the system supports REAL_TIME_TS format (0x2),
> some QPs are created with REAL_TIME_TS as ts_format, and afterwards
> RoCE gets disabled. When trying to construct a new QP, we can't use
> the qp_ts_format, that is queried from the RoCE caps table, Since it
> leads to passing 0x0 (FREE_RUNNING_TS) as the value of the qp_ts_format,
> which is different than the ts_format of the previously allocated
> QPs REAL_TIME_TS format (0x2).
> 
> Thus, to resolve this, read the sq_ts_format, which also reflect
> the supported ts format for the QP when RoCE is disabled.
> 
> Fixes: 4806f1e2fee8 ("net/mlx5: Set QP timestamp mode to default")
> Signed-off-by: Maher Sanalla <msanalla@nvidia.com>
> Signed-off-by: Or Har-Toov <ohartoov@nvidia.com>
> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>

Reviewed-by: Simon Horman <horms@kernel.org>

Re: [PATCH mlx5-next] RDMA/mlx5: Use sq timestamp as QP timestamp when RoCE is disabled

[Leon Romanovsky]

On Sun, 16 Jun 2024 19:10:36 +0300, Leon Romanovsky wrote:
> When creating a QP, one of the attributes is TS format (timestamp).
> In some devices, we have a limitation that all QPs should have the same
> ts_format. The ts_format is chosen based on the device's capability.
> The qp_ts_format cap resides under the RoCE caps table, and the
> cap will be 0 when RoCE is disabled. So when RoCE is disabled, the
> value that should be queried is sq_ts_format under HCA caps.
> 
> [...]

Applied, thanks!

[1/1] RDMA/mlx5: Use sq timestamp as QP timestamp when RoCE is disabled
      https://git.kernel.org/rdma/rdma/c/78a6cbd5145ce7

Best regards,
-- 
Leon Romanovsky <leon@kernel.org>