Re: GTE - The hardware timestamping engine

Re: GTE - The hardware timestamping engine

[Arnd Bergmann]

On Sat, Mar 20, 2021 at 12:56 PM Linus Walleij <linus.walleij@linaro.org> wrote:
>
> Hi Dipen,
>
> thanks for your mail!
>
> I involved some other kernel people to get some discussion.
> I think Kent Gibson can be of great help because he is using
> GPIOs with high precision.
>
> We actually discussed this a bit when adding support for
> realtime timestamps.

Adding Richard Cochran as well, for drivers/ptp/, he may be able to
identify whether this should be integrated into that framework in some
form.

fullquote below

> On Wed, Mar 17, 2021 at 11:29 PM Dipen Patel <dipenp@nvidia.com> wrote:
>
> > Nvidia Tegra SoCs have generic timestamping engine (GTE) hardware module which
> > can monitor SoC signals like IRQ lines and GPIO lines for state change, upon
> > detecting the change, it can timestamp and store in its internal hardware FIFO.
> > The advantage of the GTE module can be realized in applications like robotics
> > or autonomous vehicle where it can help record events with precise timestamp.
>
> That sounds very useful.
>
> Certainly the kernel shall be able to handle this.
>
> > ============
> > For GPIO:
> > ============
> > 1.  GPIO has to be configured as input and IRQ must be enabled.
> > 2.  Ask GPIO controller driver to set corresponding timestamp bit in the
> >     specified GPIO config register.
> > 3.  Translate GPIO specified by the client to its internal bitmap.
> > 3.a For example, If client specifies GPIO line 31, it could be bit 13 of GTE
> >     register.
> > 4.  Set internal bits to enable monitoring in GTE module
> > 5.  Additionally GTE driver can open up lanes for the user space application
> >     as a client and can send timestamping events directly to the application.
>
> I have some concerns:
>
> 1. GPIO should for all professional applications be used with the character
> device /dev/gpiochipN, under no circumstances shall the old sysfs
> ABI be used for this. In this case it is necessary because the
> character device provides events in a FIFO to userspace, which is
> what we need.
>
> The timestamp provided to userspace is an opaque 64bit
> unsigned value. I suppose we assume it is monotonic but
> you can actually augment the semantics for your specific
> stamp, as long as 64 bits is gonna work.
>
> 2. The timestamp for the chardev is currently obtained in
> drivers/gpio/gpiolib-cdev.c like this:
>
> static u64 line_event_timestamp(struct line *line)
> {
>         if (test_bit(FLAG_EVENT_CLOCK_REALTIME, &line->desc->flags))
>                 return ktime_get_real_ns();
>
>         return ktime_get_ns();
> }
>
> What you want to do is to add a new flag for hardware timestamps
> and use that if available. FLAG_EVENT_CLOCK_HARDWARE?
> FLAG_EVENT_CLOCK_NATIVE?
>
> Then you need to figure out a mechanism so we can obtain
> the right timestamp from the hardware event right here,
> you can hook into the GPIO driver if need be, we can
> figure out the gpio_chip for a certain line for sure.
>
> So you first need to augment the userspace
> ABI and the character device code to add this. See
> commit 26d060e47e25f2c715a1b2c48fea391f67907a30
> "gpiolib: cdev: allow edge event timestamps to be configured as REALTIME"
> by Kent Gibson to see what needs to be done.
>
> 3. Also patch tools/gpio/gpio-event-mon.c to support this flag and use that
> for prototyping and proof of concept.
>
> > ============
> > For IRQ:
> > ============
>
> Marc Zyngier and/or Thomas Gleixner know this stuff.
>
> It does make sense to add some infrastructure so that GPIO events
> and IRQs can use the same timestamping hardware.
>
> And certainly you will also want to use this timestamp for
> IIO devices? If it is just GPIOs and IRQs today, it will be
> gyroscopes and accelerometers tomorrow, am I right?
>
> Yours,
> Linus Walleij

Re: GTE - The hardware timestamping engine

[Richard Cochran]

On Sat, Mar 20, 2021 at 01:44:20PM +0100, Arnd Bergmann wrote:
> Adding Richard Cochran as well, for drivers/ptp/, he may be able to
> identify whether this should be integrated into that framework in some
> form.

I'm not familiar with the GTE, but it sounds like it is a (free
running?) clock with time stamping inputs.  If so, then it could
expose a PHC.  That gets you functionality:

- clock_gettime() and friends
- comparison ioctl between GTE clock and CLOCK_REALTIME
- time stamping channels with programmable input selection

The mentioned applications (robotics and autonomous vehicle, so near
and dear to my heart) surely already use the PHC API for dealing with
network and system time sources, and so exposing the GTE as a PHC
means that user space programs will have a consistent API.

[ The only drawback I can see is the naming of the C language
  identifiers in include/uapi/linux/ptp_clock.h.  If that bothers
  people, then these can be changed to something more generic while
  keeping compatibility aliases. ]

Thanks,
Richard

Re: GTE - The hardware timestamping engine

[Dipen Patel]

Hi Richard,

Thanks for your input and time. Please see below follow up.

On 3/20/21 8:38 AM, Richard Cochran wrote:
> On Sat, Mar 20, 2021 at 01:44:20PM +0100, Arnd Bergmann wrote:
>> Adding Richard Cochran as well, for drivers/ptp/, he may be able to
>> identify whether this should be integrated into that framework in some
>> form.
> 
> I'm not familiar with the GTE, but it sounds like it is a (free
> running?) clock with time stamping inputs.  If so, then it could
> expose a PHC.  That gets you functionality:
> 
> - clock_gettime() and friends
> - comparison ioctl between GTE clock and CLOCK_REALTIME
> - time stamping channels with programmable input selection
> 
GTE gets or rather records the timestamps from the TSC
(timestamp system coutner) so its not attached to GTE as any
one can access TSC, so not sure if we really need to implement PHC
and/or clock_* and friends for the GTE. I believe burden to find correlation
between various clock domains should be on the clients, consider below
example.

Networking client has access to both PTP and GTE, it would be its job
to find the correlations if that is at all needed based on whatever
use case that client serves. GTE in above may come in picture if said client
has some GPIO configured and wants timestamp on it.

Sorry if I misunderstood anything, you can elaborate more as I am also
interested in how GTE can fit in PTP framework and which usecase it can
help doing so.

> The mentioned applications (robotics and autonomous vehicle, so near
> and dear to my heart) surely already use the PHC API for dealing with
> network and system time sources, and so exposing the GTE as a PHC
> means that user space programs will have a consistent API.
> 
> [ The only drawback I can see is the naming of the C language
>   identifiers in include/uapi/linux/ptp_clock.h.  If that bothers
>   people, then these can be changed to something more generic while
>   keeping compatibility aliases. ]
> 
> Thanks,
> Richard
> 

Thanks,
Best Regards,
Dipen Patel

Re: GTE - The hardware timestamping engine

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1908 bytes --]

On Mon, Mar 22, 2021 at 01:33:38PM -0700, Dipen Patel wrote:
> Hi Richard,
> 
> Thanks for your input and time. Please see below follow up.
> 
> On 3/20/21 8:38 AM, Richard Cochran wrote:
> > On Sat, Mar 20, 2021 at 01:44:20PM +0100, Arnd Bergmann wrote:
> >> Adding Richard Cochran as well, for drivers/ptp/, he may be able to
> >> identify whether this should be integrated into that framework in some
> >> form.
> > 
> > I'm not familiar with the GTE, but it sounds like it is a (free
> > running?) clock with time stamping inputs.  If so, then it could
> > expose a PHC.  That gets you functionality:
> > 
> > - clock_gettime() and friends
> > - comparison ioctl between GTE clock and CLOCK_REALTIME
> > - time stamping channels with programmable input selection
> > 
> GTE gets or rather records the timestamps from the TSC
> (timestamp system coutner) so its not attached to GTE as any
> one can access TSC, so not sure if we really need to implement PHC
> and/or clock_* and friends for the GTE. I believe burden to find correlation
> between various clock domains should be on the clients, consider below
> example.

I agree. My understanding is the the TSC is basically an SoC-wide clock
that can be (and is) used by several hardware blocks. There's an
interface for software to read out the value, but it's part of a block
called TKE (time-keeping engine, if I recall correctly) that implements
various clock sources and watchdog functionality.

As a matter of fact, I recall typing up a driver for that at some point
but I don't recall if I ever sent it out or what became of it. I can't
find it upstream at least.

Anyway, I think given that the GTE doesn't provide that clock itself but
rather just a means of taking a snapshot of that clock and stamping
certain events with that, it makes more sense to provide that clock from
the TKE driver.

Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GTE - The hardware timestamping engine

[Richard Cochran]

On Tue, Mar 23, 2021 at 10:03:18AM +0100, Thierry Reding wrote:
> I agree. My understanding is the the TSC is basically an SoC-wide clock
> that can be (and is) used by several hardware blocks. There's an
> interface for software to read out the value, but it's part of a block
> called TKE (time-keeping engine, if I recall correctly) that implements
> various clock sources and watchdog functionality.

...

> Anyway, I think given that the GTE doesn't provide that clock itself but
> rather just a means of taking a snapshot of that clock and stamping
> certain events with that, it makes more sense to provide that clock from
> the TKE driver.

It sounds like TKE + GTE together act like a PHC, and GTE doesn't
need/want its own SW interface.

Thanks,
Richard