[RFC] Video events, version 2.2

[RFC] Video events, version 2.2

[Sakari Ailus]

Hi,


Here's the version 2.2 of the video events RFC. It's based on Laurent
Pinchart's original RFC and versions 2 and 2.1 which I wrote. The old 
RFC is available here:

<URL:http://www.spinics.net/lists/linux-media/msg11056.html>

Added Mauro to Cc.

Changes to version 2.1
--------------------

V4L2_EVENT_ALL is now 0 instead 0x07ffffff.

V4L2_EVENT_RESERVED is gone. A note will be added not to use four 
topmost bits.

It's V4L2_EVENT_PRIVATE_START, not V4L2_EVENT_PRIVATE.

Interface description
---------------------

Event type is either a standard event or private event. Standard events
will be defined in videodev2.h. Private event types begin from
V4L2_EVENT_PRIVATE_START. The four topmost bits of the type should not 
be used for the moment.

#define V4L2_EVENT_ALL			0
#define V4L2_EVENT_PRIVATE_START	0x08000000

VIDIOC_DQEVENT is used to get events. count is number of pending events
after the current one. sequence is the event type sequence number and
the data is specific to event type.

The user will get the information that there's an event through
exception file descriptors by using select(2). When an event is
available the poll handler sets POLLPRI which wakes up select. -EINVAL
will be returned if there are no pending events.

VIDIOC_SUBSCRIBE_EVENT and VIDIOC_UNSUBSCRIBE_EVENT are used to
subscribe and unsubscribe from events. The argument is struct
v4l2_event_subscription which now only contains the type field for the
event type. Every event can be subscribed or unsubscribed by one ioctl
by using special type V4L2_EVENT_ALL.


struct v4l2_event {
	__u32		count;
	__u32		type;
	__u32		sequence;
	struct timeval	timestamp;
	__u32		reserved[8];
	__u8		data[64];
};

struct v4l2_event_subscription {
	__u32		type;
	__u32		reserved[8];
};

#define VIDIOC_DQEVENT		_IOR('V', 84, struct v4l2_event)
#define VIDIOC_SUBSCRIBE_EVENT	_IOW('V', 85, struct
				     v4l2_event_subscription)
#define VIDIOC_UNSUBSCRIBE_EVENT _IOW('V', 86, struct
				      v4l2_event_subscription)


The size of the event queue is decided by the driver. Which events will
be discarded on queue overflow depends on the implementation.


Questions
---------

None on my side.

Comments and questions are still very very welcome.

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: [RFC] Video events, version 2.2

[Ivan T. Ivanov]

Hi Sakari, 

On Fri, 2009-10-23 at 13:18 +0300, Sakari Ailus wrote:
> Hi,
> 
> 
> Here's the version 2.2 of the video events RFC. It's based on Laurent
> Pinchart's original RFC and versions 2 and 2.1 which I wrote. The old 
> RFC is available here:
> 
> <URL:http://www.spinics.net/lists/linux-media/msg11056.html>
> 
> Added Mauro to Cc.
> 
> Changes to version 2.1
> --------------------
> 
> V4L2_EVENT_ALL is now 0 instead 0x07ffffff.
> 
> V4L2_EVENT_RESERVED is gone. A note will be added not to use four 
> topmost bits.
> 
> It's V4L2_EVENT_PRIVATE_START, not V4L2_EVENT_PRIVATE.
> 
> Interface description
> ---------------------
> 
> Event type is either a standard event or private event. Standard events
> will be defined in videodev2.h. Private event types begin from
> V4L2_EVENT_PRIVATE_START. The four topmost bits of the type should not 
> be used for the moment.
> 
> #define V4L2_EVENT_ALL			0
> #define V4L2_EVENT_PRIVATE_START	0x08000000
> 
> VIDIOC_DQEVENT is used to get events. count is number of pending events
> after the current one. sequence is the event type sequence number and
> the data is specific to event type.
> 
> The user will get the information that there's an event through
> exception file descriptors by using select(2). When an event is
> available the poll handler sets POLLPRI which wakes up select. -EINVAL
> will be returned if there are no pending events.
> 
> VIDIOC_SUBSCRIBE_EVENT and VIDIOC_UNSUBSCRIBE_EVENT are used to
> subscribe and unsubscribe from events. The argument is struct
> v4l2_event_subscription which now only contains the type field for the
> event type. Every event can be subscribed or unsubscribed by one ioctl
> by using special type V4L2_EVENT_ALL.
> 
> 
> struct v4l2_event {
> 	__u32		count;
> 	__u32		type;
> 	__u32		sequence;
> 	struct timeval	timestamp;

Can we use 'struct timespec' here. This will force actual 
implementation to use high-resolution source if possible, 
and remove hundreds gettimeofday() in user space, which 
should be used for event synchronization, with more 
power friendly clock_getres(CLOCK_MONOTONIC).

Thank you.

iivanov


> 	__u32		reserved[8];
> 	__u8		data[64];
> };
> 
> struct v4l2_event_subscription {
> 	__u32		type;
> 	__u32		reserved[8];
> };
> 
> #define VIDIOC_DQEVENT		_IOR('V', 84, struct v4l2_event)
> #define VIDIOC_SUBSCRIBE_EVENT	_IOW('V', 85, struct
> 				     v4l2_event_subscription)
> #define VIDIOC_UNSUBSCRIBE_EVENT _IOW('V', 86, struct
> 				      v4l2_event_subscription)
> 
> 
> The size of the event queue is decided by the driver. Which events will
> be discarded on queue overflow depends on the implementation.
> 
> 
> Questions
> ---------
> 
> None on my side.
> 
> Comments and questions are still very very welcome.
> 

Re: [RFC] Video events, version 2.2

[Sakari Ailus]

Ivan T. Ivanov wrote:
> Hi Sakari, 

Hi,

> On Fri, 2009-10-23 at 13:18 +0300, Sakari Ailus wrote:
[clip]
>> struct v4l2_event {
>> 	__u32		count;
>> 	__u32		type;
>> 	__u32		sequence;
>> 	struct timeval	timestamp;
> 
> Can we use 'struct timespec' here. This will force actual 
> implementation to use high-resolution source if possible, 
> and remove hundreds gettimeofday() in user space, which 
> should be used for event synchronization, with more 
> power friendly clock_getres(CLOCK_MONOTONIC).

Good point. I originally picked timeval since it was used in 
v4l2_buffer. The spec tells to use gettimeofday() for system time but 
clock skewing is causes problems in video encoding. 
clock_getres(CLOCK_MONOTONIC) is free of clock skewing and thus should 
be more suitable for this kind of use.

I also propose to use timespec instead of timeval.

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: [RFC] Video events, version 2.2

[Hans Verkuil]

On Saturday 24 October 2009 23:56:24 Sakari Ailus wrote:
> Ivan T. Ivanov wrote:
> > Hi Sakari, 
> 
> Hi,
> 
> > On Fri, 2009-10-23 at 13:18 +0300, Sakari Ailus wrote:
> [clip]
> >> struct v4l2_event {
> >> 	__u32		count;
> >> 	__u32		type;
> >> 	__u32		sequence;
> >> 	struct timeval	timestamp;
> > 
> > Can we use 'struct timespec' here. This will force actual 
> > implementation to use high-resolution source if possible, 
> > and remove hundreds gettimeofday() in user space, which 
> > should be used for event synchronization, with more 
> > power friendly clock_getres(CLOCK_MONOTONIC).
> 
> Good point. I originally picked timeval since it was used in 
> v4l2_buffer. The spec tells to use gettimeofday() for system time but 
> clock skewing is causes problems in video encoding. 
> clock_getres(CLOCK_MONOTONIC) is free of clock skewing and thus should 
> be more suitable for this kind of use.
> 
> I also propose to use timespec instead of timeval.
> 

Hi Sakari,

What is that status of the event API? It is my impression that it is pretty
much finished. Sakari, can you make a final 2.3 RFC? Then Guru can take over
and start the implementation.

Regards,

	Hans

-- 
Hans Verkuil - video4linux developer - sponsored by TANDBERG Telecom

Re: [RFC] Video events, version 2.2

[Sakari Ailus]

Hans Verkuil wrote:
> On Saturday 24 October 2009 23:56:24 Sakari Ailus wrote:
>> Ivan T. Ivanov wrote:
>>> Hi Sakari, 
>> Hi,
>>
>>> On Fri, 2009-10-23 at 13:18 +0300, Sakari Ailus wrote:
>> [clip]
>>>> struct v4l2_event {
>>>> 	__u32		count;
>>>> 	__u32		type;
>>>> 	__u32		sequence;
>>>> 	struct timeval	timestamp;
>>> Can we use 'struct timespec' here. This will force actual 
>>> implementation to use high-resolution source if possible, 
>>> and remove hundreds gettimeofday() in user space, which 
>>> should be used for event synchronization, with more 
>>> power friendly clock_getres(CLOCK_MONOTONIC).
>> Good point. I originally picked timeval since it was used in 
>> v4l2_buffer. The spec tells to use gettimeofday() for system time but 
>> clock skewing is causes problems in video encoding. 
>> clock_getres(CLOCK_MONOTONIC) is free of clock skewing and thus should 
>> be more suitable for this kind of use.
>>
>> I also propose to use timespec instead of timeval.
>>
> 
> Hi Sakari,
> 
> What is that status of the event API? It is my impression that it is pretty
> much finished. Sakari, can you make a final 2.3 RFC? Then Guru can take over
> and start the implementation.

Ah.

One thing that I was still wondering was that are there use cases where 
other kind of time stamps might be useful? I guess that when the V4L2 
was designed no-one though of the need for time stamps of different 
type. So are there use cases where gettimeofday() style stamps would 
still be better?

In that case we might choose to leave it driver's decision to decide 
what kind of timestamps to use and in that case application would just 
have to know. The alternative would be to use union and a flag telling 
what's in there.

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: [RFC] Video events, version 2.2

[Hans Verkuil]

On Wednesday 11 November 2009 18:29:52 Sakari Ailus wrote:
> Hans Verkuil wrote:
> > On Saturday 24 October 2009 23:56:24 Sakari Ailus wrote:
> >> Ivan T. Ivanov wrote:
> >>> Hi Sakari, 
> >> Hi,
> >>
> >>> On Fri, 2009-10-23 at 13:18 +0300, Sakari Ailus wrote:
> >> [clip]
> >>>> struct v4l2_event {
> >>>> 	__u32		count;
> >>>> 	__u32		type;
> >>>> 	__u32		sequence;
> >>>> 	struct timeval	timestamp;
> >>> Can we use 'struct timespec' here. This will force actual 
> >>> implementation to use high-resolution source if possible, 
> >>> and remove hundreds gettimeofday() in user space, which 
> >>> should be used for event synchronization, with more 
> >>> power friendly clock_getres(CLOCK_MONOTONIC).
> >> Good point. I originally picked timeval since it was used in 
> >> v4l2_buffer. The spec tells to use gettimeofday() for system time but 
> >> clock skewing is causes problems in video encoding. 
> >> clock_getres(CLOCK_MONOTONIC) is free of clock skewing and thus should 
> >> be more suitable for this kind of use.
> >>
> >> I also propose to use timespec instead of timeval.
> >>
> > 
> > Hi Sakari,
> > 
> > What is that status of the event API? It is my impression that it is pretty
> > much finished. Sakari, can you make a final 2.3 RFC? Then Guru can take over
> > and start the implementation.
> 
> Ah.
> 
> One thing that I was still wondering was that are there use cases where 
> other kind of time stamps might be useful? I guess that when the V4L2 
> was designed no-one though of the need for time stamps of different 
> type. So are there use cases where gettimeofday() style stamps would 
> still be better?

If you ever need to relate an event to a specific captured frame, then that
might well be useful. But I can't think of an actual use case, though.

> In that case we might choose to leave it driver's decision to decide 
> what kind of timestamps to use and in that case application would just 
> have to know. The alternative would be to use union and a flag telling 
> what's in there.
> 

Let's go with timespec. If we need to add an event that has to relate to
a specific captured frame then it is always possible to add a struct timeval
as part of the event data for that particular event.

Regards,

	Hans

-- 
Hans Verkuil - video4linux developer - sponsored by TANDBERG Telecom

Re: [RFC] Video events, version 2.2

[Mauro Carvalho Chehab]

Em Wed, 11 Nov 2009 18:59:09 +0100
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> On Wednesday 11 November 2009 18:29:52 Sakari Ailus wrote:
> > Hans Verkuil wrote:
> > > On Saturday 24 October 2009 23:56:24 Sakari Ailus wrote:
> > >> Ivan T. Ivanov wrote:
> > >>> Hi Sakari, 
> > >> Hi,
> > >>
> > >>> On Fri, 2009-10-23 at 13:18 +0300, Sakari Ailus wrote:
> > >> [clip]
> > >>>> struct v4l2_event {
> > >>>> 	__u32		count;
> > >>>> 	__u32		type;
> > >>>> 	__u32		sequence;
> > >>>> 	struct timeval	timestamp;
> > >>> Can we use 'struct timespec' here. This will force actual 
> > >>> implementation to use high-resolution source if possible, 
> > >>> and remove hundreds gettimeofday() in user space, which 
> > >>> should be used for event synchronization, with more 
> > >>> power friendly clock_getres(CLOCK_MONOTONIC).
> > >> Good point. I originally picked timeval since it was used in 
> > >> v4l2_buffer. The spec tells to use gettimeofday() for system time but 
> > >> clock skewing is causes problems in video encoding. 
> > >> clock_getres(CLOCK_MONOTONIC) is free of clock skewing and thus should 
> > >> be more suitable for this kind of use.
> > >>
> > >> I also propose to use timespec instead of timeval.
> > >>
> > > 
> > > Hi Sakari,
> > > 
> > > What is that status of the event API? It is my impression that it is pretty
> > > much finished. Sakari, can you make a final 2.3 RFC? Then Guru can take over
> > > and start the implementation.
> > 
> > Ah.
> > 
> > One thing that I was still wondering was that are there use cases where 
> > other kind of time stamps might be useful? I guess that when the V4L2 
> > was designed no-one though of the need for time stamps of different 
> > type. So are there use cases where gettimeofday() style stamps would 
> > still be better?
> 
> If you ever need to relate an event to a specific captured frame, then that
> might well be useful. But I can't think of an actual use case, though.
> 
> > In that case we might choose to leave it driver's decision to decide 
> > what kind of timestamps to use and in that case application would just 
> > have to know. The alternative would be to use union and a flag telling 
> > what's in there.
> > 
> 
> Let's go with timespec. If we need to add an event that has to relate to
> a specific captured frame then it is always possible to add a struct timeval
> as part of the event data for that particular event.

I don't agree. It is better to use the same timestamp type used by the streaming
interface. Having two different ways to represent it for the same devices is
confusing, and changing it later doesn't make sense. I foresee some cases where
correlating the two timestamps would be a need.
> Regards,
> 
> 	Hans
> 

Cheers,
Mauro

Re: [RFC] Video events, version 2.2

[Hans Verkuil]

> Em Wed, 11 Nov 2009 18:59:09 +0100
> Hans Verkuil <hverkuil@xs4all.nl> escreveu:
>
>> On Wednesday 11 November 2009 18:29:52 Sakari Ailus wrote:
>> > Hans Verkuil wrote:
>> > > On Saturday 24 October 2009 23:56:24 Sakari Ailus wrote:
>> > >> Ivan T. Ivanov wrote:
>> > >>> Hi Sakari,
>> > >> Hi,
>> > >>
>> > >>> On Fri, 2009-10-23 at 13:18 +0300, Sakari Ailus wrote:
>> > >> [clip]
>> > >>>> struct v4l2_event {
>> > >>>> 	__u32		count;
>> > >>>> 	__u32		type;
>> > >>>> 	__u32		sequence;
>> > >>>> 	struct timeval	timestamp;
>> > >>> Can we use 'struct timespec' here. This will force actual
>> > >>> implementation to use high-resolution source if possible,
>> > >>> and remove hundreds gettimeofday() in user space, which
>> > >>> should be used for event synchronization, with more
>> > >>> power friendly clock_getres(CLOCK_MONOTONIC).
>> > >> Good point. I originally picked timeval since it was used in
>> > >> v4l2_buffer. The spec tells to use gettimeofday() for system time
>> but
>> > >> clock skewing is causes problems in video encoding.
>> > >> clock_getres(CLOCK_MONOTONIC) is free of clock skewing and thus
>> should
>> > >> be more suitable for this kind of use.
>> > >>
>> > >> I also propose to use timespec instead of timeval.
>> > >>
>> > >
>> > > Hi Sakari,
>> > >
>> > > What is that status of the event API? It is my impression that it is
>> pretty
>> > > much finished. Sakari, can you make a final 2.3 RFC? Then Guru can
>> take over
>> > > and start the implementation.
>> >
>> > Ah.
>> >
>> > One thing that I was still wondering was that are there use cases
>> where
>> > other kind of time stamps might be useful? I guess that when the V4L2
>> > was designed no-one though of the need for time stamps of different
>> > type. So are there use cases where gettimeofday() style stamps would
>> > still be better?
>>
>> If you ever need to relate an event to a specific captured frame, then
>> that
>> might well be useful. But I can't think of an actual use case, though.
>>
>> > In that case we might choose to leave it driver's decision to decide
>> > what kind of timestamps to use and in that case application would just
>> > have to know. The alternative would be to use union and a flag telling
>> > what's in there.
>> >
>>
>> Let's go with timespec. If we need to add an event that has to relate to
>> a specific captured frame then it is always possible to add a struct
>> timeval
>> as part of the event data for that particular event.
>
> I don't agree. It is better to use the same timestamp type used by the
> streaming
> interface. Having two different ways to represent it for the same devices
> is
> confusing, and changing it later doesn't make sense. I foresee some cases
> where
> correlating the two timestamps would be a need.

No, I don't mean that we change it later. The timespec will be used as the
timestamp for all events. But if we need to add an event type that needs
to be correlated with a buffer, then it can add the timeval as part of the
event-specific data. Actually, if you need to relate it to a buffer, then
you might be better off by specifying the buffer index directly.

Normally I would prefer to go with just a single timeval as well, but
gettimeofday() is quite expensive (I've noticed that before), and for all
the events that I can think of right now it is not needed.

Does gettimeofday() also have additional penalties on embedded systems? If
I understand the original comment correctly, then gettimeofday() is less
power-friendly than clock_getres. Why is that? Is that just because
gettimeofday() needs more CPU cycles or are there additional reasons?

Regards,

       Hans

>> Regards,
>>
>> 	Hans
>>
>
> Cheers,
> Mauro
>


-- 
Hans Verkuil - video4linux developer - sponsored by TANDBERG Telecom

Re: [RFC] Video events, version 2.2

[Eino-Ville Talvala]

On 11/13/2009 8:00 AM, Hans Verkuil wrote:
>    
>> Em Wed, 11 Nov 2009 18:59:09 +0100
>> Hans Verkuil<hverkuil@xs4all.nl>  escreveu:
>>
>>      
>>> On Wednesday 11 November 2009 18:29:52 Sakari Ailus wrote:
>>>        
>>>> Hans Verkuil wrote:
>>>>          
>>>>> On Saturday 24 October 2009 23:56:24 Sakari Ailus wrote:
>>>>>            
>>>>>> Ivan T. Ivanov wrote:
>>>>>>              
>>>>>>> Hi Sakari,
>>>>>>>                
>>>>>> Hi,
>>>>>>
>>>>>>              
>>>>>>> On Fri, 2009-10-23 at 13:18 +0300, Sakari Ailus wrote:
>>>>>>>                
>>>>>> [clip]
>>>>>>              
>>>>>>>> struct v4l2_event {
>>>>>>>> 	__u32		count;
>>>>>>>> 	__u32		type;
>>>>>>>> 	__u32		sequence;
>>>>>>>> 	struct timeval	timestamp;
>>>>>>>>                  
>>>>>>> Can we use 'struct timespec' here. This will force actual
>>>>>>> implementation to use high-resolution source if possible,
>>>>>>> and remove hundreds gettimeofday() in user space, which
>>>>>>> should be used for event synchronization, with more
>>>>>>> power friendly clock_getres(CLOCK_MONOTONIC).
>>>>>>>                
>>>>>> Good point. I originally picked timeval since it was used in
>>>>>> v4l2_buffer. The spec tells to use gettimeofday() for system time
>>>>>>              
>>> but
>>>        
>>>>>> clock skewing is causes problems in video encoding.
>>>>>> clock_getres(CLOCK_MONOTONIC) is free of clock skewing and thus
>>>>>>              
>>> should
>>>        
>>>>>> be more suitable for this kind of use.
>>>>>>
>>>>>> I also propose to use timespec instead of timeval.
>>>>>>
>>>>>>              
>>>>> Hi Sakari,
>>>>>
>>>>> What is that status of the event API? It is my impression that it is
>>>>>            
>>> pretty
>>>        
>>>>> much finished. Sakari, can you make a final 2.3 RFC? Then Guru can
>>>>>            
>>> take over
>>>        
>>>>> and start the implementation.
>>>>>            
>>>> Ah.
>>>>
>>>> One thing that I was still wondering was that are there use cases
>>>>          
>>> where
>>>        
>>>> other kind of time stamps might be useful? I guess that when the V4L2
>>>> was designed no-one though of the need for time stamps of different
>>>> type. So are there use cases where gettimeofday() style stamps would
>>>> still be better?
>>>>          
>>> If you ever need to relate an event to a specific captured frame, then
>>> that
>>> might well be useful. But I can't think of an actual use case, though.
>>>
>>>        
>>>> In that case we might choose to leave it driver's decision to decide
>>>> what kind of timestamps to use and in that case application would just
>>>> have to know. The alternative would be to use union and a flag telling
>>>> what's in there.
>>>>
>>>>          
>>> Let's go with timespec. If we need to add an event that has to relate to
>>> a specific captured frame then it is always possible to add a struct
>>> timeval
>>> as part of the event data for that particular event.
>>>        
>> I don't agree. It is better to use the same timestamp type used by the
>> streaming
>> interface. Having two different ways to represent it for the same devices
>> is
>> confusing, and changing it later doesn't make sense. I foresee some cases
>> where
>> correlating the two timestamps would be a need.
>>      
> No, I don't mean that we change it later. The timespec will be used as the
> timestamp for all events. But if we need to add an event type that needs
> to be correlated with a buffer, then it can add the timeval as part of the
> event-specific data. Actually, if you need to relate it to a buffer, then
> you might be better off by specifying the buffer index directly.
>
> Normally I would prefer to go with just a single timeval as well, but
> gettimeofday() is quite expensive (I've noticed that before), and for all
> the events that I can think of right now it is not needed.
>
> Does gettimeofday() also have additional penalties on embedded systems? If
> I understand the original comment correctly, then gettimeofday() is less
> power-friendly than clock_getres. Why is that? Is that just because
> gettimeofday() needs more CPU cycles or are there additional reasons?
>
> Regards,
>
>         Hans
>
>    

I think we have a use case for events that would require correlating 
with frames, although I agree that the buffer index would be far simpler 
to match with than a timestamp.  The specific feature is letting the 
application know exactly what sensor settings were used with a given 
frame, which is essential for our slowly-developing computational camera 
API, which will be changing sensor parameters on nearly every frame 
boundary.

I think one event is probably sufficient to encode the relevant register 
values of our sensor.  Would you expect there to be any issue with 
having an event happen per frame?

Thanks,

Eino-Ville (Eddy) Talvala
Camera 2.0 Project, Computer Graphics Lab
Stanford University

Re: [RFC] Video events, version 2.2

[Sakari Ailus]

Eino-Ville Talvala wrote:
> I think we have a use case for events that would require correlating 
> with frames, although I agree that the buffer index would be far simpler 
> to match with than a timestamp.  The specific feature is letting the 
> application know exactly what sensor settings were used with a given 
> frame, which is essential for our slowly-developing computational camera 
> API, which will be changing sensor parameters on nearly every frame 
> boundary.
> 
> I think one event is probably sufficient to encode the relevant register 
> values of our sensor.  Would you expect there to be any issue with 
> having an event happen per frame?

I do expect several events per frame from the AEWB, AF and histogram 
statistics and no problems. :-)

But if I understand correctly, the registers are some kind of metadata 
associated to the frame? That perhaps includes exposure time, gain etc. 
The events interface would be good for this if the metadata fits to a 
single v4l2_event structure. A new ioctl could be an alternative, 
perhaps it could be a private ioctl first.

This is more or less comparable to the H3A statistics IMO. So the user 
space gets an event and can query the H3A data.

Associating events to a single frame is slightly troublesome since a 
succesful frame reception is only certain when it already has happened. 
There could be a metadata event and after that a receive buffer overflow 
that spoils the frame. In that case the field_count could be just 
incremented without dequeueing any buffers, though.

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: [RFC] Video events, version 2.2

[Eino-Ville Talvala]

On 11/13/2009 11:05 AM, Sakari Ailus wrote:
> Eino-Ville Talvala wrote:
>> I think we have a use case for events that would require correlating 
>> with frames, although I agree that the buffer index would be far 
>> simpler to match with than a timestamp.  The specific feature is 
>> letting the application know exactly what sensor settings were used 
>> with a given frame, which is essential for our slowly-developing 
>> computational camera API, which will be changing sensor parameters on 
>> nearly every frame boundary.
>>
>> I think one event is probably sufficient to encode the relevant 
>> register values of our sensor.  Would you expect there to be any 
>> issue with having an event happen per frame?
>
> I do expect several events per frame from the AEWB, AF and histogram 
> statistics and no problems. :-)
>
> But if I understand correctly, the registers are some kind of metadata 
> associated to the frame? That perhaps includes exposure time, gain 
> etc. The events interface would be good for this if the metadata fits 
> to a single v4l2_event structure. A new ioctl could be an alternative, 
> perhaps it could be a private ioctl first.
>
> This is more or less comparable to the H3A statistics IMO. So the user 
> space gets an event and can query the H3A data.
>
> Associating events to a single frame is slightly troublesome since a 
> succesful frame reception is only certain when it already has 
> happened. There could be a metadata event and after that a receive 
> buffer overflow that spoils the frame. In that case the field_count 
> could be just incremented without dequeueing any buffers, though.
>
Right, all of the sensor settings that applied to that particular 
frame.  We're changing the sensor settings on nearly every frame, so 
it's fairly key to keep track of them in some way, and events seem to be 
far nicer solution than what we currently do (which involves abusing the 
frame input field, as it was the fastest thing I saw to hack in).

Of course, the event queue and frame queue would have to be kept in 
sync, or just let the app discard events that apply to frames it never 
saw - as long as the event queue is a bit bigger than the frame queue, I 
don't think there'd be a problem in practice.

Eino-Ville Talvala
Camera 2.0 Project
Stanford University

Re: [RFC] Video events, version 2.2

[Sakari Ailus]

Mauro Carvalho Chehab wrote:
[clip]
>>>> Hi Sakari,
>>>>
>>>> What is that status of the event API? It is my impression that it is pretty
>>>> much finished. Sakari, can you make a final 2.3 RFC? Then Guru can take over
>>>> and start the implementation.
>>> Ah.
>>>
>>> One thing that I was still wondering was that are there use cases where 
>>> other kind of time stamps might be useful? I guess that when the V4L2 
>>> was designed no-one though of the need for time stamps of different 
>>> type. So are there use cases where gettimeofday() style stamps would 
>>> still be better?
>> If you ever need to relate an event to a specific captured frame, then that
>> might well be useful. But I can't think of an actual use case, though.
>>
>>> In that case we might choose to leave it driver's decision to decide 
>>> what kind of timestamps to use and in that case application would just 
>>> have to know. The alternative would be to use union and a flag telling 
>>> what's in there.
>>>
>> Let's go with timespec. If we need to add an event that has to relate to
>> a specific captured frame then it is always possible to add a struct timeval
>> as part of the event data for that particular event.
> 
> I don't agree. It is better to use the same timestamp type used by the streaming
> interface. Having two different ways to represent it for the same devices is
> confusing, and changing it later doesn't make sense. I foresee some cases where
> correlating the two timestamps would be a need.

timespec style timestamps are superior in video encoding, for example. 
timeval is wall clock time which is unsuitable for video encoding due to 
clock slewing and daylight saving time.

ALSA and Gstreamer use monotonic clock (someone correct me if I'm 
wrong!, cc Stefan Kost) which is more usable for multimedia 
applications. The rate for gettimeofday() clock is different than 
clock_getres(CLOCK_MONOTONIC) which in practice means that the process 
acquiring the video buffers must call clock_getres() after VIDIOC_DQBUF 
to properly timestamp the buffers. This kind of timestamping, however, 
depends on the process' ability to run immediately.

I wouldn't want to carry this kind of problems on to the event interface.

One possibility would be also to create events when new video buffers 
become available. Then both the event and the corresponding video buffer 
would be available, having the same field_count. Perhaps not pretty but 
would well make it possible to compare the timestamps.

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com