Re: [PATCH v4.1 3/3] v4l: Add V4L2_BUF_FLAG_TIMESTAMP_SOF and use it

[Laurent Pinchart <laurent.pinchart@ideasonboard.com>]

Hi Hans,

On Thursday 12 December 2013 13:37:10 Hans Verkuil wrote:
> Sakari asked me to reply to this old thread...

He asked me to reply as well :-)

> On 09/06/13 13:05, Laurent Pinchart wrote:
> > On Thursday 05 September 2013 19:31:30 Sakari Ailus wrote:
> >> On Sat, Aug 31, 2013 at 11:43:18PM +0200, Laurent Pinchart wrote:
> >>> On Friday 30 August 2013 19:08:48 Sakari Ailus wrote:
> >>>> On Fri, Aug 30, 2013 at 01:31:44PM +0200, Laurent Pinchart wrote:
> >>>>> On Thursday 29 August 2013 14:33:39 Sakari Ailus wrote:
> >>>>>> On Thu, Aug 29, 2013 at 01:25:05AM +0200, Laurent Pinchart wrote:
> >>>>>>> On Wednesday 28 August 2013 19:39:19 Sakari Ailus wrote:
> >>>>>>>> On Wed, Aug 28, 2013 at 06:14:44PM +0200, Laurent Pinchart
> >>>>>>>> wrote:
> >>>>>>>> ...
> >>>>>>>> 
> >>>>>>>>>>> UVC devices timestamp frames when the frame is captured,
> >>>>>>>>>>> not when the first pixel is transmitted.
> >>>>>>>>>> 
> >>>>>>>>>> I.e. we shouldn't set the SOF flag? "When the frame is
> >>>>>>>>>> captured" doesn't say much, or almost anything in terms of
> >>>>>>>>>> *when*. The frames have exposure time and rolling shutter
> >>>>>>>>>> makes a difference, too.
> >>>>>>>>> 
> >>>>>>>>> The UVC 1.1 specification defines the timestamp as
> >>>>>>>>> 
> >>>>>>>>> "The source clock time in native deviceclock units when the
> >>>>>>>>> raw frame capture begins."
> >>>>>>>>> 
> >>>>>>>>> What devices do in practice may differ :-)
> >>>>>>>> 
> >>>>>>>> I think that this should mean start-of-frame - exposure time.
> >>>>>>>> I'd really wonder if any practical implementation does that
> >>>>>>>> however.
> >>>>>>> 
> >>>>>>> It's start-of-frame - exposure time - internal delays (UVC webcams
> >>>>>>> are supposed to report their internal delay value as well).
> >>>>>> 
> >>>>>> Do they report it? How about the exposure time?
> >>>>> 
> >>>>> It's supposed to be configurable.
> >>>> 
> >>>> Is the exposure reported with the frame so it could be used to
> >>>> construct
> >>>> the per-frame SOF timestamp?
> >>> 
> >>> Not when auto-exposure is turned on I'm afraid :-S
> >>> 
> >>> I believe that the capture timestamp makes more sense than the SOF
> >>> timestamp for applications. SOF/EOF are more of a poor man's timestamp
> >>> in case nothing else is available, but when you want to synchronize
> >>> multiple audio and/or video streams the capture timestamp is what you're
> >>> interested in. I don't think converting a capture timestamp to an SOF
> >>> would be a good idea.
> >> 
> >> I'm not quite sure of that --- I think the SOF/EOF will be more stable
> >> than the exposure start which depends on the exposure time. If you're
> >> recording a video you may want to keep the time between the frames
> >> constant.
> > 
> > I can see two main use cases for timestamps. The first one is multi-stream
> > synchronization (audio and video, stereo video, ...), the second one is
> > playback rate control.
> > 
> > To synchronize media streams you need to timestamp samples with a common
> > clock. Timestamps must be correlated to the time at which the sound and/or
> > image events occur. If we consider the speed of sound and speed of light
> > as negligible (the former could be compensated for if needed, but that's
> > out of scope), the time at which the sound or image is produced can be
> > considered as equal to the time at which they're captured. Given that we
> > only need to synchronize streams here, an offset wouldn't matter, so any
> > clock that is synchronized to the capture clock with a fixed offset would
> > do. The SOF event, in particular, will do if the capture time and device
> > processing time is constant, and if interrupt latencies are kept small
> > enough.. So will the EOF event if the transmission time is also constant.
> > 
> > Granted, frames are not captured at a precise point of time, as the sensor
> > needs to be exposed for a certain duration. There is thus no such thing as
> > a capture time, we instead have a capture interval. However, that's
> > irrelevant for multi-video synchronization purposes. It could matter for
> > audio+video synchronization though.
> > 
> > Regarding playback rate control, the goal is to render frames at the same
> > rate they are captured. If the frame rate isn't constant (for instance
> > because of a variable exposure time), then a time stamp is required for
> > every frame. Here we care about the difference between timestamps for two
> > consecutive frames, and the start of capture timestamp is what will give
> > best results.
> > 
> > Let's consider three frames, A, B and C, captured as follows.
> > 
> > 
> > 00000000001111111111222222222233333333334444444444555555555566666666667777
> > 01234567890123456789012345678901234567890123456789012345678901234567890123
> > 
> > | --------- A ------------ |      | ----- B ----- |      | ----- C ----- |
> > 
> > On the playback side, we want to display A for a duration of 34. If we
> > timestamp the frames with the start of capture time, we will have the
> > following timestamps.
> > 
> > A  0
> > B  34
> > C  57
> > 
> > B-A = 34, which is the time during which A needs to be displayed.
> > 
> > If we use the end of capture time, we will get
> > 
> > A  27
> > B  50
> > C  73
> > 
> > B-A = 23, which is too short.
> > 
> >> Nevertheless --- if we don't get such a timestamp from the device this
> >> will only remain speculation. Applications might be best using e.g. half
> >> the frame period to get a guesstimate of the differences between the two
> >> timestamps.
> > 
> > Obviously if the device can't provide the start of capture timestamp we
> > will need to use any source of timestamps, but I believe we should aim
> > for start of capture as a first class citizen.
> > 
> >>>>>> If you know them all you can calculate the SOF timestamp. The fewer
> >>>>>> timestamps are available for user programs the better.
> >>>>>> 
> >>>>>> It's another matter then if there are webcams that report these
> >>>>>> values wrong.
> >>>>> 
> >>>>> There most probably are :-)
> >>>>> 
> >>>>>> Then you could get timestamps that are complete garbage. But I guess
> >>>>>> you could compare them to the current monotonic timestamp and detect
> >>>>>> such cases.
> >>>>>> 
> >>>>>>>> What's your suggestion; should we use the SOF flag for this or
> >>>>>>>> do you prefer the end-of-frame timestamp instead? I think it'd
> >>>>>>>> be quite nice for drivers to know which one is which without
> >>>>>>>> having to guess, and based on the above start-of-frame comes as
> >>>>>>>> close to that definition as is meaningful.
> >>>>>>> 
> >>>>>>> SOF is better than EOF. Do we need a start-of-capture flag, or
> >>>>>>> could we document SOF as meaning start-of-capture or start-of-
> >>>>>>> reception depending on what the device can do ?
> >>>>>> 
> >>>>>> One possibility is to dedicate a few flags for this; by using three
> >>>>>> bits we'd get eight different timestamps already. But I have to say
> >>>>>> that fewer is better. :-)
> >>>>> 
> >>>>> Does it really need to be a per-buffer flag ? This seems to be a
> >>>>> driver-wide (or at least device-wide) behaviour to me.
> >>>> 
> >>>> Same goes for timestamp clock sources. It was concluded to use buffer
> >>>> flags for those as well.
> >>> 
> >>> Yes, and I don't think I was convinced, so I'm not convinced here either
> >>> 
> >>> :-)
> >>>
> >>>> Using a control for the purpose would however require quite non-zero
> >>>> amount of initialisation code from each driver so that would probably
> >>>> need to be sorted out first.
> >>> 
> >>> We could also use a capabilities flag.
> >> 
> >> Interesting idea. I'm fine that as well. Hans?
> 
> That would work for uvc, but not in the general case. Depending on the video
> routing you might have either SOF or EOF timestamps. Unlikely, I admit, but
> I feel keeping this flag in v4l2_buffers is the most generic solution.

My main concern about this (beside using an extra buffer flags bit, which is a 
scarce resource - but OK, that's not really a big concern) is complexity for 
userspace. Correctly handling buffer timestamps when the timestamp type can 
vary per buffer isn't easy, and I most applications will likely implement it 
wrong. I expect most applications to look at the timestamp type of the first 
buffer and use that information for all subsequent buffers. This would defeat 
the point of having per-buffer timestamp types.

-- 
Regards,

Laurent Pinchart