Re: [PATCH for-next V1 00/11] Add completion timestamping support

Steve Wise
2015-05-28T21:41:58Z

On 5/21/2015 9:56 AM, Or Gerlitz wrote:
> Hi Doug,
>
> This patchset adds completion timestamping supports for verbs consumers.
>
> Timestamping is used by applications in order to know when a WQE was
> received/transmitted by the HW. The value is given is HCA hardware cycles,
> but could be easily converted as the hardware's core clock frequecny is
> available through extension of query device.
>
> Moreover, we add an ability to read the HCA's current clock. This could be
> useful on order to synchronize events to the wall clock.
>
> This functionality is achieved by adding/extending the following verbs:
>
> create_cq - create_cq is extended in order to allow passing creation flags
> to the CQ creation function. We change IB/core --> vendors API
> to be easily extendible by passing a struct which contains
> comp_vectors, cqe and the new flags parameter. In order to create
> CQ which supports timestamping, IB_CQ_FLAGS_TIMESTAMP should be given.
>
> query_device - We extend query_device uverb further by giving the hardware's
> clock frequency and the timestamp mask (the number of timestamp
> bits which are supported). If timestamp isn't supported, 0 is returned.
>
> In order to read the timestamp in the WQE, the user needs to query the device
> for support, create an appropriate CQ (using the extanded uverb with
> IB_CQ_FLAGS_TIMESTAMP) and poll the CQ with an extended poll_cq verb (currently,
> only implemented in user-space).
>
> In mlx4, allowing the user to read the core clock efficiently involves mapping
> this area of the hardware to user-space (being done by using a mmap command)
> and reading the clock from the correct offset of the page.
>
> This offset is returned in the vendor's specific data from mlx4's kernel driver
> to the mlx4's user-space driver. query_device is modified in order to support
> passing this vendor specific data. A user-space application could use a new
> verb in order to read the hardware's clock.
>
> Translating the hardware's clock into ms could be done by dividing this
> value by hca_core_clock (which is returned by the extended version of
> query_device uverb).
>
> A user-space application could get the current HW's clock by executing
>
> ibv_query_values_ex(struct ibv_context *context, uint32_t q_values,
> struct ibv_values_ex *values)
>
> The function gets a mask of the values to query and return their values.
> Vendors could either implement this as a uverb command or use their
> user-space driver to return those values directly from the HW (the mlx4 way).

I'm just reviewing this now, and I haven't looked at the user side, but
it appears the CQE timestamp is available for all devices to support or
not in a generic manner. IE it is part of the extended CQ UAPI. But
the task of fetching the current timestamp value/mask seems to be
device-specific. Shouldn't that also be a standard operation that
devices can choose to support or not? IE an application can
generically setup a CQ and get timestamps on CQEs, but it seems the
application would have to have device-specific code to get the current
timestamp/mask. Perhaps I'm not understanding the design?

Steve.