[RFC] Monotonic clock usage in buffer timestamps

[RFC] Monotonic clock usage in buffer timestamps

[Laurent Pinchart]

Hi everybody,

The V4L2 specification documents the v4l2_buffer timestamp field as

"For input streams this is the system time (as returned by the gettimeofday() 
function) when the first data byte was captured."

The system time is a pretty bad clock source to timestamp buffers, as it can 
jump back and forth in time. Using a monotonic clock, as returned by 
clock_gettime(CLOCK_MONOTONIC) (or ktime_get_ts() in the kernel), would be 
much more useful.

Several drivers already use a monotonic clock instead of the system clock, 
which currently violates the V4L2 specification. As those drivers do the right 
thing from a technical point of view, I'd really hate "fixing" them by making 
them use gettimeofday().

We should instead fix the V4L2 specification to mandate the use of a monotonic 
clock (which could then also support hardware timestamps when they are 
available). Would such a change be acceptable ?

-- 
Regards,

Laurent Pinchart

Re: [RFC] Monotonic clock usage in buffer timestamps

[Rémi Denis-Courmont]

   Hello,



On Tue, 1 Nov 2011 13:24:35 +0100, Laurent Pinchart

<laurent.pinchart@ideasonboard.com> wrote:

> We should instead fix the V4L2 specification to mandate the use of a

> monotonic clock (which could then also support hardware timestamps when

> they are available). Would such a change be acceptable ?



I'd rather have the real time clock everywhere, than a driver-dependent

clock, if it comes to that.

Nevertheless, I agree that the monotonic clock is better than the real

time clock.

In user space, VLC, Gstreamer already switched to monotonic a while ago as

far as I know.



And I guess there is no way to detect this, other than detect ridiculously

large gap between the timestamp and the current clock value?



-- 

Rémi Denis-Courmont

http://www.remlab.net/

Re: [RFC] Monotonic clock usage in buffer timestamps

[Laurent Pinchart]

Hi Rémi,

On Tuesday 01 November 2011 13:36:50 Rémi Denis-Courmont wrote:
> On Tue, 1 Nov 2011 13:24:35 +0100, Laurent Pinchart wrote:
> > We should instead fix the V4L2 specification to mandate the use of a
> > monotonic clock (which could then also support hardware timestamps when
> > they are available). Would such a change be acceptable ?
> 
> I'd rather have the real time clock everywhere, than a driver-dependent
> clock, if it comes to that.

That's my opinion as well. Modifying drivers to use a monotonic clock is easy, 
and I can provide patches. The real issue is whether this can be accepted, as 
it would change the spec.

> Nevertheless, I agree that the monotonic clock is better than the real
> time clock.
> In user space, VLC, Gstreamer already switched to monotonic a while ago as
> far as I know.
> 
> And I guess there is no way to detect this, other than detect ridiculously
> large gap between the timestamp and the current clock value?

That's right. We could add a device capability flag if needed, but that 
wouldn't help older applications that expect system time in the timestamps.

-- 
Regards,

Laurent Pinchart

Re: [RFC] Monotonic clock usage in buffer timestamps

[Daniel Glöckner]

Hello,

On Tue, Nov 01, 2011 at 01:49:46PM +0100, Laurent Pinchart wrote:
> > Nevertheless, I agree that the monotonic clock is better than the real
> > time clock.
> > In user space, VLC, Gstreamer already switched to monotonic a while ago as
> > far as I know.
> > 
> > And I guess there is no way to detect this, other than detect ridiculously
> > large gap between the timestamp and the current clock value?
> 
> That's right. We could add a device capability flag if needed, but that 
> wouldn't help older applications that expect system time in the timestamps.

I just so happen to have tried to use V4L2 and ALSA timestamps in a
single application. In ALSA the core supports switching between
monotonic and realtime timestamps, with the library always using
monotonic available.

How about making all drivers record monotonic timestamps and doing
the conversion to/from realtime timestamps in v4l2-ioctl.c's
__video_do_ioctl if requested? We then just need an extension of the
spec to switch to monotonic, which can be implemented without touching
a single driver.

  Daniel

Re: [RFC] Monotonic clock usage in buffer timestamps

[Sakari Ailus]

On Tue, Nov 01, 2011 at 01:24:35PM +0100, Laurent Pinchart wrote:
> Hi everybody,
> 
> The V4L2 specification documents the v4l2_buffer timestamp field as
> 
> "For input streams this is the system time (as returned by the gettimeofday() 
> function) when the first data byte was captured."
> 
> The system time is a pretty bad clock source to timestamp buffers, as it can 
> jump back and forth in time. Using a monotonic clock, as returned by 
> clock_gettime(CLOCK_MONOTONIC) (or ktime_get_ts() in the kernel), would be 
> much more useful.
> 
> Several drivers already use a monotonic clock instead of the system clock, 
> which currently violates the V4L2 specification. As those drivers do the right 
> thing from a technical point of view, I'd really hate "fixing" them by making 
> them use gettimeofday().
> 
> We should instead fix the V4L2 specification to mandate the use of a monotonic 
> clock (which could then also support hardware timestamps when they are 
> available). Would such a change be acceptable ?

I'm in favour of that. I don't think wall clock timestamps are really useful
to begin with. If you really need them, you can always do gettimeofday() in
the user space.

For any kind of a/v synchronisation where precise timestamps matter the
monotonic clock is the way to go.

Cheers,

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [RFC] Monotonic clock usage in buffer timestamps

[Sakari Ailus]

On Wed, Nov 02, 2011 at 10:10:46AM +0100, Daniel Glöckner wrote:
> Hello,

Hi Daniel,

> On Tue, Nov 01, 2011 at 01:49:46PM +0100, Laurent Pinchart wrote:
> > > Nevertheless, I agree that the monotonic clock is better than the real
> > > time clock.
> > > In user space, VLC, Gstreamer already switched to monotonic a while ago as
> > > far as I know.
> > > 
> > > And I guess there is no way to detect this, other than detect ridiculously
> > > large gap between the timestamp and the current clock value?
> > 
> > That's right. We could add a device capability flag if needed, but that 
> > wouldn't help older applications that expect system time in the timestamps.
> 
> I just so happen to have tried to use V4L2 and ALSA timestamps in a
> single application. In ALSA the core supports switching between
> monotonic and realtime timestamps, with the library always using
> monotonic available.
> 
> How about making all drivers record monotonic timestamps and doing
> the conversion to/from realtime timestamps in v4l2-ioctl.c's
> __video_do_ioctl if requested? We then just need an extension of the
> spec to switch to monotonic, which can be implemented without touching
> a single driver.

Converting between the two can be done when making the timestamp but it's
non-trivial at other times and likely isn't supported. I could be wrong,
though. This might lead to e.g. timestamps that are taken before switching
to summer time and for which the conversion is done after the switch. This
might be a theoretical possibility, but there might be also unfavourable
interaction with the NTP.

I'd probably rather just make a new timestamp in wall clock time in
v4l2-ioctl.c if needed using do_gettimeofday(). It also needs to be agreed
how does the user space request that to be done.

Or just do the wall clock timestamps user space as they are typically
critical in timing.

How would this work for you?

Kind regards,

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [RFC] Monotonic clock usage in buffer timestamps

[Laurent Pinchart]

Hi Sakari,

On Wednesday 02 November 2011 11:14:49 Sakari Ailus wrote:
> On Wed, Nov 02, 2011 at 10:10:46AM +0100, Daniel Glöckner wrote:
> > On Tue, Nov 01, 2011 at 01:49:46PM +0100, Laurent Pinchart wrote:
> > > > Nevertheless, I agree that the monotonic clock is better than the
> > > > real time clock.
> > > > In user space, VLC, Gstreamer already switched to monotonic a while
> > > > ago as far as I know.
> > > > 
> > > > And I guess there is no way to detect this, other than detect
> > > > ridiculously large gap between the timestamp and the current clock
> > > > value?
> > > 
> > > That's right. We could add a device capability flag if needed, but that
> > > wouldn't help older applications that expect system time in the
> > > timestamps.
> > 
> > I just so happen to have tried to use V4L2 and ALSA timestamps in a
> > single application. In ALSA the core supports switching between
> > monotonic and realtime timestamps, with the library always using
> > monotonic available.
> > 
> > How about making all drivers record monotonic timestamps and doing
> > the conversion to/from realtime timestamps in v4l2-ioctl.c's
> > __video_do_ioctl if requested? We then just need an extension of the
> > spec to switch to monotonic, which can be implemented without touching
> > a single driver.
> 
> Converting between the two can be done when making the timestamp but it's
> non-trivial at other times and likely isn't supported. I could be wrong,
> though. This might lead to e.g. timestamps that are taken before switching
> to summer time and for which the conversion is done after the switch. This
> might be a theoretical possibility, but there might be also unfavourable
> interaction with the NTP.
> 
> I'd probably rather just make a new timestamp in wall clock time in
> v4l2-ioctl.c if needed using do_gettimeofday(). It also needs to be agreed
> how does the user space request that to be done.
> 
> Or just do the wall clock timestamps user space as they are typically
> critical in timing.
> 
> How would this work for you?

I agree that converting between the timestamps is error prone. Compatibility 
code, if required, should probably just call do_gettimeofday() in v4l2_ioctl.c 
(or possibly in videobuf2 ?).

The real problem would be to decide how to select compatibility mode. Adding a 
flag so that applications can request monotonic timestamps is doable, but 
would open the door to a long transition period, and I'd like to avoid that.

I expect most applications to not even notice the switch from the system clock 
to a monotonic clock. Using system timestamps for video buffers is unreliable 
and broken by design, so we could argue that applications that would break (if 
any) are already broken :-)

-- 
Regards,

Laurent Pinchart

Re: [RFC] Monotonic clock usage in buffer timestamps

[Daniel Glöckner]

On Wed, Nov 02, 2011 at 12:14:49PM +0200, Sakari Ailus wrote:
> > How about making all drivers record monotonic timestamps and doing
> > the conversion to/from realtime timestamps in v4l2-ioctl.c's
> > __video_do_ioctl if requested? We then just need an extension of the
> > spec to switch to monotonic, which can be implemented without touching
> > a single driver.
> 
> Converting between the two can be done when making the timestamp but it's
> non-trivial at other times and likely isn't supported. I could be wrong,
> though. This might lead to e.g. timestamps that are taken before switching
> to summer time and for which the conversion is done after the switch. This
> might be a theoretical possibility, but there might be also unfavourable
> interaction with the NTP.

Summertime/wintertime is purely a userspace thing. UTC as returned by
gettimeofday is unaffected by that.

NTP AFAIK adjusts the speed of the monotonic clock, so there is a constant
delta between wall clock time and clock monotonic unless there is a leap
second or someone calls settimeofday. Applications currently using the
wall clock timestamps should have trouble dealing with that as well.

> I'd probably rather just make a new timestamp in wall clock time in
> v4l2-ioctl.c if needed using do_gettimeofday().

I think that would be worse than subtracting ktime_get_monotonic_offset().
You don't know the delay between capturing a frame and calling dqbuf.

If there is a settimeofday between capturing a frame and calling dqbuf,
the wall time clock was probably wrong when the frame was captured
and subtracting the new ktime_get_monotonic_offset() yields a better
timestamp.

> Or just do the wall clock timestamps user space as they are typically
> critical in timing.
>
> How would this work for you?

As I keep the cpu busy with video encoding in the same thread, I'd expect
a high jitter from taking timestamps in userspace.

  Daniel

Re: [RFC] Monotonic clock usage in buffer timestamps

[Rémi Denis-Courmont]

On Wed, 2 Nov 2011 11:58:04 +0100, Daniel Glöckner <daniel-gl@gmx.net>

wrote:

>> Converting between the two can be done when making the timestamp but

it's

>> non-trivial at other times and likely isn't supported. I could be

wrong,

>> though. This might lead to e.g. timestamps that are taken before

>> switching

>> to summer time and for which the conversion is done after the switch.

>> This might be a theoretical possibility, but there might be also

>> unfavourable interaction with the NTP.

> 

> Summertime/wintertime is purely a userspace thing. UTC as returned by

> gettimeofday is unaffected by that.



Right, DST is a non-issue.



> NTP AFAIK adjusts the speed of the monotonic clock, so there is a

constant

> delta between wall clock time and clock monotonic



For NTP it depends. Simple NTP, as in ntpdate, warps the wall clock.

Full-blown NTP only adjusts the speed.



> unless there is a leap

> second or someone calls settimeofday. Applications currently using the

> wall clock timestamps should have trouble dealing with that as well.



I can think of at least three other sources of wall clock time, that could

trigger a warp:

 - GPS receiver (TAI),

 - cellular modem (NITZ),

 - and, of course, manual setting.



So if at all possible I'd much prefer monotonic over real timestamps.



-- 

Rémi Denis-Courmont

http://www.remlab.net/

Re: [RFC] Monotonic clock usage in buffer timestamps

[Sakari Ailus]

On Wed, Nov 02, 2011 at 12:14:49PM +0200, Sakari Ailus wrote:
...
> Or just do the wall clock timestamps user space as they are typically
> critical in timing.

There was a rather important "not" missing in this sentence.

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [RFC] Monotonic clock usage in buffer timestamps

[Sakari Ailus]

On Wed, Nov 02, 2011 at 11:58:04AM +0100, Daniel Glöckner wrote:
> On Wed, Nov 02, 2011 at 12:14:49PM +0200, Sakari Ailus wrote:
> > > How about making all drivers record monotonic timestamps and doing
> > > the conversion to/from realtime timestamps in v4l2-ioctl.c's
> > > __video_do_ioctl if requested? We then just need an extension of the
> > > spec to switch to monotonic, which can be implemented without touching
> > > a single driver.
> > 
> > Converting between the two can be done when making the timestamp but it's
> > non-trivial at other times and likely isn't supported. I could be wrong,
> > though. This might lead to e.g. timestamps that are taken before switching
> > to summer time and for which the conversion is done after the switch. This
> > might be a theoretical possibility, but there might be also unfavourable
> > interaction with the NTP.
> 
> Summertime/wintertime is purely a userspace thing. UTC as returned by
> gettimeofday is unaffected by that.

Indeed, that's correct.

> NTP AFAIK adjusts the speed of the monotonic clock, so there is a constant
> delta between wall clock time and clock monotonic unless there is a leap
> second or someone calls settimeofday. Applications currently using the
> wall clock timestamps should have trouble dealing with that as well.

I wonder if applications do use it for something these days. Some might, but
I don't know of any that would be affected.

> > I'd probably rather just make a new timestamp in wall clock time in
> > v4l2-ioctl.c if needed using do_gettimeofday().
> 
> I think that would be worse than subtracting ktime_get_monotonic_offset().
> You don't know the delay between capturing a frame and calling dqbuf.

Right. As Laurent suggested, doing that in videobuf2 is better option since
it gets called when the buffer gets dequeueable.

If videobuf2 already has knowledge on what kind of timestamp the user
expects it would be possible to do it here.

> If there is a settimeofday between capturing a frame and calling dqbuf,
> the wall time clock was probably wrong when the frame was captured
> and subtracting the new ktime_get_monotonic_offset() yields a better
> timestamp.
> 
> > Or just do the wall clock timestamps user space as they are typically
> > critical in timing.
> >
> > How would this work for you?
> 
> As I keep the cpu busy with video encoding in the same thread, I'd expect
> a high jitter from taking timestamps in userspace.

True.

Kind regards,

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk