RFC: use of timestamp/sequence in v4l2_buffer

RFC: use of timestamp/sequence in v4l2_buffer

[Hans Verkuil]

Hi all,

During the Media Workshop last week we discussed how the timestamp and sequence
fields in struct v4l2_buffer should be used.

While trying to document the exact behavior I realized that there are a few
missing pieces.

Open questions with regards to the sequence field:

1) Should the first frame that was captured or displayed after starting streaming
for the first time always start with sequence index 0? In my opinion it should.

2) Should the sequence counter be reset to 0 after a STREAMOFF? Or should it only
be reset to 0 after REQBUFS/CREATE_BUFS is called?

3) Should the sequence counter behave differently for mem2mem devices? With the
current definition both the capture and display sides of a mem2mem device just have
their own independent sequence counter.

4) If frames are skipped, should the sequence counter skip as well? In my opinion
it shouldn't.

5) Should the sequence counter always be monotonically increasing? I think it should.

Open questions with regards to the timestamp field:

6) For output devices the timestamp field can be used to determine when to transmit
the frame. In practice there are no output drivers that support this. It is also
unclear how this would work: if the timestamp is 1 hour into the future, should the
driver hold on to that frame for one hour? If another frame is queued with a timestamp
that's earlier than the previous frame, should that frame be output first?

I am inclined to drop this behavior from the spec. Should we get drivers that actually
implement this, then we need to clarify the spec and add a new capability flag somewhere
to tell userspace that you can actually use the timestamp for this purpose.

7) Should the timestamp field always be monotonically increasing? Or it is possible
to get timestamps that jump around? This makes sense for encoders that create B-frames
referring to frames captured earlier than an I-frame.

8) How should the timestamp field be handled for mem2mem devices? Setting the timestamp
is pointless for the display side of a mem2mem device (as discussed above as well).
One option is that the mem2mem driver sets the timestamp when it starts processing a
queued frame. This timestamp is returned when the buffer is dequeued. This timestamp
is also copied to the processed frame (on the capture side of the mem2mem device),
thus allowing one to relate the captured processed frame to the source frame.

Even if encoder devices rearrange the frames (as discussed in the previous point),
the timestamp can still be used to map the frames.

Comments are welcome!

Regards,

	Hans

Re: RFC: use of timestamp/sequence in v4l2_buffer

[Rémi Denis-Courmont]

Le mardi 4 septembre 2012 13:38:06, Hans Verkuil a écrit :
> 7) Should the timestamp field always be monotonically increasing? Or it is
> possible to get timestamps that jump around? This makes sense for encoders
> that create B-frames referring to frames captured earlier than an I-frame.

I would expect an encoder to output frames in DTS order rather than PTS and a 
decoder to output frames in PTS order. The timestamp field is the only 
indication of the PCR, so an application might not recover if the timestamp 
jumps backward.

If there is an ambiguity between PTS and DTS, I think it should be documented 
and specified per format what the timestamp is.

-- 
Rémi Denis-Courmont
http://www.remlab.net/

Re: RFC: use of timestamp/sequence in v4l2_buffer

[Sakari Ailus]

Hi Hans,

Thanks for the RFC!

On Tue, Sep 04, 2012 at 12:38:06PM +0200, Hans Verkuil wrote:
> Hi all,
> 
> During the Media Workshop last week we discussed how the timestamp and
> sequence fields in struct v4l2_buffer should be used.
> 
> While trying to document the exact behavior I realized that there are a
> few missing pieces.
> 
> Open questions with regards to the sequence field:
> 
> 1) Should the first frame that was captured or displayed after starting
> streaming for the first time always start with sequence index 0? In my
> opinion it should.

I agree.

> 2) Should the sequence counter be reset to 0 after a STREAMOFF? Or should
> it only be reset to 0 after REQBUFS/CREATE_BUFS is called?

Definitely not after CREATE_BUFS. Streaming may be ongoing when the IOCTL is
called, and this will cause a great deal of trouble. I'd propose resetting
it to zero at streamon (or streamoff) time.

> 3) Should the sequence counter behave differently for mem2mem devices?
> With the current definition both the capture and display sides of a
> mem2mem device just have their own independent sequence counter.
> 
> 4) If frames are skipped, should the sequence counter skip as well? In my
> opinion it shouldn't.

Do you mean skipping incrementing the counter, or skipping the frame number?
:-) In my opinion the sequence number must be increased in this case. Not
doing so would make it difficult to figure out frames have been skipped in
the first place. That may be important in some cases. I can't think of any
adverse effects this could result; the OMAP 3 ISP driver does so, for
example.

On the side of additional positive effects, consider the following quite
realistic scenario: A frame is skipped on a single buffer queue of a device
producing two streams of the same source, e.g. a sensor, whereas no frame is
skipped on the second video buffer queue. Not incrementing the sequence
would make the sequence numbers out-of-sync, and the situation would even be
difficult to detect from the user space --- frame sequence numbers are
important in associating buffers from different streams to the same original
captured image.

> 5) Should the sequence counter always be monotonically increasing? I think
> it should.

With the exception of the above, in my opinion yes. I.e. you're not supposed
to have a decrease in field_count until it wraps around.

> Open questions with regards to the timestamp field:
> 
> 6) For output devices the timestamp field can be used to determine when to
> transmit the frame. In practice there are no output drivers that support
> this. It is also unclear how this would work: if the timestamp is 1 hour
> into the future, should the driver hold on to that frame for one hour? If
> another frame is queued with a timestamp that's earlier than the previous
> frame, should that frame be output first?
> 
> I am inclined to drop this behavior from the spec. Should we get drivers
> that actually implement this, then we need to clarify the spec and add a
> new capability flag somewhere to tell userspace that you can actually use
> the timestamp for this purpose.
> 
> 7) Should the timestamp field always be monotonically increasing? Or it is
> possible to get timestamps that jump around? This makes sense for encoders
> that create B-frames referring to frames captured earlier than an I-frame.

And for decoders that need to hold a key frame longer than others. (Or to
save system memory resources, return it to the user space with the proposed
READ_ONLY flag not agreed on yet.)

Cheers,

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	XMPP: sailus@retiisi.org.uk

Re: RFC: use of timestamp/sequence in v4l2_buffer

[Laurent Pinchart]

Hi,

On Thursday 13 September 2012 21:44:48 Sakari Ailus wrote:
> On Tue, Sep 04, 2012 at 12:38:06PM +0200, Hans Verkuil wrote:
> > Hi all,
> > 
> > During the Media Workshop last week we discussed how the timestamp and
> > sequence fields in struct v4l2_buffer should be used.
> > 
> > While trying to document the exact behavior I realized that there are a
> > few missing pieces.
> > 
> > Open questions with regards to the sequence field:
> > 
> > 1) Should the first frame that was captured or displayed after starting
> > streaming for the first time always start with sequence index 0? In my
> > opinion it should.
> 
> I agree.

Agreed.

> > 2) Should the sequence counter be reset to 0 after a STREAMOFF? Or should
> > it only be reset to 0 after REQBUFS/CREATE_BUFS is called?
> 
> Definitely not after CREATE_BUFS. Streaming may be ongoing when the IOCTL is
> called, and this will cause a great deal of trouble. I'd propose resetting
> it to zero at streamon (or streamoff) time.

Agreed.

> > 3) Should the sequence counter behave differently for mem2mem devices?
> > With the current definition both the capture and display sides of a
> > mem2mem device just have their own independent sequence counter.

They need to be independent, as one frame can be encoded/decoded to several 
frames (or the other way around).

> > 4) If frames are skipped, should the sequence counter skip as well? In my
> > opinion it shouldn't.
> 
> Do you mean skipping incrementing the counter, or skipping the frame number?
> :-) In my opinion the sequence number must be increased in this case. Not
> doing so would make it difficult to figure out frames have been skipped in
> the first place. That may be important in some cases. I can't think of any
> adverse effects this could result; the OMAP 3 ISP driver does so, for
> example.

I agree, I think the sequence counter should be incremented for skipped 
frames. Not all drivers can detect frame skips though, or how many frames are 
skipped.

> On the side of additional positive effects, consider the following quite
> realistic scenario: A frame is skipped on a single buffer queue of a device
> producing two streams of the same source, e.g. a sensor, whereas no frame is
> skipped on the second video buffer queue. Not incrementing the sequence
> would make the sequence numbers out-of-sync, and the situation would even
> be difficult to detect from the user space --- frame sequence numbers are
> important in associating buffers from different streams to the same
> original captured image.
> 
> > 5) Should the sequence counter always be monotonically increasing? I think
> > it should.
> 
> With the exception of the above, in my opinion yes. I.e. you're not supposed
> to have a decrease in field_count until it wraps around.
> 
> > Open questions with regards to the timestamp field:
> > 
> > 6) For output devices the timestamp field can be used to determine when to
> > transmit the frame. In practice there are no output drivers that support
> > this. It is also unclear how this would work: if the timestamp is 1 hour
> > into the future, should the driver hold on to that frame for one hour? If
> > another frame is queued with a timestamp that's earlier than the previous
> > frame, should that frame be output first?
> > 
> > I am inclined to drop this behavior from the spec. Should we get drivers
> > that actually implement this, then we need to clarify the spec and add a
> > new capability flag somewhere to tell userspace that you can actually use
> > the timestamp for this purpose.
> > 
> > 7) Should the timestamp field always be monotonically increasing? Or it is
> > possible to get timestamps that jump around? This makes sense for encoders
> > that create B-frames referring to frames captured earlier than an I-frame.
> 
> And for decoders that need to hold a key frame longer than others. (Or to
> save system memory resources, return it to the user space with the proposed
> READ_ONLY flag not agreed on yet.)

-- 
Regards,

Laurent Pinchart