Re: [PATCH for-next V1 00/11] Add completion timestamping support

[Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>]

On 5/21/2015 9:56 AM, Or Gerlitz wrote:
> Hi Doug,
>
> This patchset adds completion timestamping supports for verbs consumers.
>
> Timestamping is used by applications in order to know when a WQE was
> received/transmitted by the HW. The value is given is HCA hardware cycles,
> but could be easily converted as the hardware's core clock frequecny is
> available through extension of query device.
>
> Moreover, we add an ability to read the HCA's current clock. This could be
> useful on order to synchronize events to the wall clock.
>
> This functionality is achieved by adding/extending the following verbs:
>
> create_cq - create_cq is extended in order to allow passing creation flags
> to the CQ creation function. We change IB/core --> vendors API
> to be easily extendible by passing a struct which contains
> comp_vectors, cqe and the new flags parameter. In order to create
> CQ which supports timestamping, IB_CQ_FLAGS_TIMESTAMP should be given.
>
> query_device - We extend query_device uverb further by giving the hardware's
> clock frequency and the timestamp mask (the number of timestamp
> bits which are supported). If timestamp isn't supported, 0 is returned.
>
> In order to read the timestamp in the WQE, the user needs to query the device
> for support, create an appropriate CQ (using the extanded uverb with
> IB_CQ_FLAGS_TIMESTAMP) and poll the CQ with an extended poll_cq verb (currently,
> only implemented in user-space).
>
> In mlx4, allowing the user to read the core clock efficiently involves mapping
> this area of the hardware to user-space (being done by using a mmap command)
> and reading the clock from the correct offset of the page.
>
> This offset is returned in the vendor's specific data from mlx4's kernel driver
> to the mlx4's user-space driver. query_device is modified in order to support
> passing this vendor specific data. A user-space application could use a new
> verb in order to read the hardware's clock.
>
> Translating the hardware's clock into ms could be done by dividing this
> value by hca_core_clock (which is returned by the extended version of
> query_device uverb).
>
> A user-space application could get the current HW's clock by executing
>
> ibv_query_values_ex(struct ibv_context *context, uint32_t q_values,
>                      struct ibv_values_ex *values)
>
> The function gets a mask of the values to query and return their values.
> Vendors could either implement this as a uverb command or use their
> user-space driver to return those values directly from the HW (the mlx4 way).

I'm just reviewing this now, and I haven't looked at the user side, but 
it appears the CQE timestamp is available for all devices to support or 
not in a generic manner.  IE it is part of the extended CQ UAPI.  But 
the task of fetching the current timestamp value/mask seems to be 
device-specific.   Shouldn't that also be a standard operation that 
devices can choose to support or not?    IE an application can 
generically setup a CQ and get timestamps on CQEs, but it seems the 
application would have to have device-specific code to get the current 
timestamp/mask.  Perhaps I'm not understanding the design?

Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html