Re: [dpdk-dev] [RFC PATCH 1/3] net/mlx5: add timestamp-to-ns converter from libibverbs

[Olivier Matz <olivier.matz@6wind.com>]

Hi Vivien,

On Thu, Jun 25, 2020 at 03:01:17PM -0400, Vivien Didelot wrote:
> From: Patrick Keroulas <patrick.keroulas@radio-canada.ca>
> 
> While some devices update their own clock info to provide current time,
> mlx5dv part of libibverbs already handles this and also converts any
> raw counter cycle to nanoseconds.
> 
> Signed-off-by: Patrick Keroulas <patrick.keroulas@radio-canada.ca>
> ---
>  drivers/common/mlx5/linux/mlx5_glue.c   | 16 +++++++++++++
>  drivers/common/mlx5/linux/mlx5_glue.h   |  4 ++++
>  drivers/net/mlx5/linux/mlx5_ethdev_os.c | 30 +++++++++++++++++++++++++
>  drivers/net/mlx5/linux/mlx5_os.c        |  1 +
>  drivers/net/mlx5/mlx5.h                 |  1 +
>  5 files changed, 52 insertions(+)
> 
> diff --git a/drivers/common/mlx5/linux/mlx5_glue.c b/drivers/common/mlx5/linux/mlx5_glue.c
> index c91ee33bb..cac24015b 100644
> --- a/drivers/common/mlx5/linux/mlx5_glue.c
> +++ b/drivers/common/mlx5/linux/mlx5_glue.c
> @@ -80,6 +80,20 @@ mlx5_glue_query_rt_values_ex(struct ibv_context *context,
>  	return ibv_query_rt_values_ex(context, values);
>  }
>  
> +static int
> +mlx5_glue_get_clock_info(struct ibv_context *context,
> +			  struct mlx5dv_clock_info *clock_info)
> +{
> +	return mlx5dv_get_clock_info(context, clock_info);
> +}
> +
> +static uint64_t
> +mlx5_glue_mlx5dv_ts_to_ns(struct mlx5dv_clock_info *clock_info,
> +			  uint64_t device_timestamp)
> +{
> +	return mlx5dv_ts_to_ns(clock_info, device_timestamp);
> +}
> +
>  static int
>  mlx5_glue_query_port(struct ibv_context *context, uint8_t port_num,
>  		     struct ibv_port_attr *port_attr)
> @@ -1207,6 +1221,8 @@ const struct mlx5_glue *mlx5_glue = &(const struct mlx5_glue) {
>  	.query_device = mlx5_glue_query_device,
>  	.query_device_ex = mlx5_glue_query_device_ex,
>  	.query_rt_values_ex = mlx5_glue_query_rt_values_ex,
> +	.get_clock_info = mlx5_glue_get_clock_info,
> +	.convert_ts_to_ns = mlx5_glue_mlx5dv_ts_to_ns,
>  	.query_port = mlx5_glue_query_port,
>  	.create_comp_channel = mlx5_glue_create_comp_channel,
>  	.destroy_comp_channel = mlx5_glue_destroy_comp_channel,
> diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h
> index 5d238a40a..8d05e7398 100644
> --- a/drivers/common/mlx5/linux/mlx5_glue.h
> +++ b/drivers/common/mlx5/linux/mlx5_glue.h
> @@ -123,6 +123,10 @@ struct mlx5_glue {
>  			  struct ibv_port_attr *port_attr);
>  	struct ibv_comp_channel *(*create_comp_channel)
>  		(struct ibv_context *context);
> +	int (*get_clock_info)(struct ibv_context *context,
> +			       struct mlx5dv_clock_info *clock_info);
> +	uint64_t (*convert_ts_to_ns)(struct mlx5dv_clock_info *clock_info,
> +			  uint64_t device_timestamp);
>  	int (*destroy_comp_channel)(struct ibv_comp_channel *channel);
>  	struct ibv_cq *(*create_cq)(struct ibv_context *context, int cqe,
>  				    void *cq_context,
> diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> index ab47cb531..86bec111b 100644
> --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
> @@ -355,6 +355,36 @@ mlx5_set_flags(struct rte_eth_dev *dev, unsigned int keep, unsigned int flags)
>  	return mlx5_ifreq(dev, SIOCSIFFLAGS, &request);
>  }
>  
> +/**
> + * Convert raw clock counter to nanoseconds
> + *
> + * @param dev
> + *   Pointer to Ethernet device structure.
> + * @param[in&out] timestamp
> + *   Pointer to the timestamp to be converted.
> + *
> + * @return
> + *   0 if the clock has correctly been read
> + *   The value of errno in case of error
> + */
> +int
> +mlx5_convert_ts_to_ns(struct rte_eth_dev *dev, uint64_t *timestamp)
> +{
> +	struct mlx5_priv *priv = dev->data->dev_private;
> +	struct ibv_context *ctx = priv->sh->ctx;
> +	struct mlx5dv_clock_info clock_info;
> +
> +	int err = mlx5_glue->get_clock_info(ctx, &clock_info);
> +	if (err != 0) {
> +		DRV_LOG(WARNING, "Could not get the clock info!");
> +		return err;
> +	}
> +
> +	*timestamp = mlx5_glue->convert_ts_to_ns(&clock_info, *timestamp);
> +
> +	return err;
> +}

Should mlx5dv_get_clock_info() be called for each convesion? In the man
page [1], we can read this:

       If the clock_info becomes too old then time conversion will return
       wrong conversion results.  The user must ensure that
       mlx5dv_get_clock_info(3) is called at least once every
       max_clock_info_update_nsec as returned by the mlx5dv_query_device(3)
       function.

I understand this as mlx5dv_get_clock_info() being slower than
mlx5dv_ts_to_ns(), and should not be called for each packet.

I suppose that when rdma-core library does a conversion to nanosec, it
takes in account the fact that system and device are not synchronized
(are they?).

[1] https://man7.org/linux/man-pages/man3/mlx5dv_get_clock_info.3.html

Olivier