Re: [PATCH for-next 09/10] IB/mlx4: Add timestamp_mask and hca_core_clock to query_device

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Thu, May 28, 2015 at 01:30:52PM -0500, Christoph Lameter wrote:
> On Thu, 28 May 2015, Jason Gunthorpe wrote:
> 
> > > This is sufficient since it can be converted to ns or whatever one wants.
> >
> > Sure it is sufficient, but is it a robust UAPI, will it support
> > multiple hardware vendors?
> 
> What would prevent other hardware vendors from exporting their counters?

Well, I'm not a HW vendor, so I don't know for sure, the proposal is:

timestamp_mask - how many bits are valid in the timestamp.
		 timestamp values could be 64bits the most.
hca_core_clock - timestamp is given in HW cycles, hca_core_clock
                 is the frequency of the HCA and is necessary in
		 order to convert cycles to seconds.

So presumably cycle difference is:
  (a-b) & mask
And to ns is
 a*1E9/hca_core_clock

So first, the above is a tricky bit of math to open code, integer
overflow needs to be avoided - so at a minimum I'd like to see
libibverbs provide this as a function call. I'd probably also say
difference should be an inline too.

If it is a function call then maybe we don't need to tell the app
about hca_core_clock?

Second: What about wrap around? Does it even make sense to expose less
than 64 bits to userspace? Should the driver manage wrap around to
create a flat 64 bit space?

Otherwise, if the app has to do it, there is no event indicating wrap
has occured.

Without managing wrapping, cycle count difference is not a reliable
calculation.

> > Is anyone else in ethernet using verbs to deliver IP packets?
> 
> This is not only for Ethernet. Internally Infiniband is frequently used
> and there also timestamps are useful.

Sure, but should any other vendors be commenting on this UAPI?

> Well no. There has been a history of putting time corrections etc etc into
> these. Once you move from a raw counter to actual time various
> complications may need to be dealt with. For simple time differentials the
> counter is sufficient. If you really want proper "time" when something
> happens then you may want to scale and correct etc the value and have some
> sort of time sync approach.

Sure OK, I'm sold, export a cycle counter of some sort.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html