Re: [PATCH for-next 09/10] IB/mlx4: Add timestamp_mask and hca_core_clock to query_device

Jason Gunthorpe
2015-05-28T19:50:34Z

On Thu, May 28, 2015 at 01:30:52PM -0500, Christoph Lameter wrote:
> On Thu, 28 May 2015, Jason Gunthorpe wrote:
>
> > > This is sufficient since it can be converted to ns or whatever one wants.
> >
> > Sure it is sufficient, but is it a robust UAPI, will it support
> > multiple hardware vendors?
>
> What would prevent other hardware vendors from exporting their counters?

Well, I'm not a HW vendor, so I don't know for sure, the proposal is:

timestamp_mask - how many bits are valid in the timestamp.
timestamp values could be 64bits the most.
hca_core_clock - timestamp is given in HW cycles, hca_core_clock
is the frequency of the HCA and is necessary in
order to convert cycles to seconds.

So presumably cycle difference is:
(a-b) & mask
And to ns is
a*1E9/hca_core_clock

So first, the above is a tricky bit of math to open code, integer
overflow needs to be avoided - so at a minimum I'd like to see
libibverbs provide this as a function call. I'd probably also say
difference should be an inline too.

If it is a function call then maybe we don't need to tell the app
about hca_core_clock?

Second: What about wrap around? Does it even make sense to expose less
than 64 bits to userspace? Should the driver manage wrap around to
create a flat 64 bit space?

Otherwise, if the app has to do it, there is no event indicating wrap
has occured.

Without managing wrapping, cycle count difference is not a reliable
calculation.

> > Is anyone else in ethernet using verbs to deliver IP packets?
>
> This is not only for Ethernet. Internally Infiniband is frequently used
> and there also timestamps are useful.

Sure, but should any other vendors be commenting on this UAPI?

> Well no. There has been a history of putting time corrections etc etc into
> these. Once you move from a raw counter to actual time various
> complications may need to be dealt with. For simple time differentials the
> counter is sufficient. If you really want proper "time" when something
> happens then you may want to scale and correct etc the value and have some
> sort of time sync approach.

Sure OK, I'm sold, export a cycle counter of some sort.

Jason