RFC: API to query webcams for various webcam specific properties

RFC: API to query webcams for various webcam specific properties

[Hans de Goede]

Hi All,

With libv4l giving us the ability to do some much needed image conversion in 
userspace, it has become clear that for (some) webcams more processing then 
just format conversion is necessary.

So far I've been keeping various proposed patches for doing things like 
software white balance correction out of libv4l as I first want a proper API 
for drivers to signal they need this to libv4l.

Part of the problem is that various cams needs various additional processing 
steps for best results, and currently there is no way to ask a driver which 
additional steps should be done (and using which values). Another part is that 
we do not have a complete picture of all possible existing processing steps we 
want to do, so what ever we come up with needs to be extensible.

To give an idea, here are a few things which libv4l should know about an video 
input source:
-does this cam need software whitebalance
-does this cam need software auto exposure
-does this cam need gamma correction, and what initial gamma to use
-if the sensor is mounted upside down, and the hardware cannot flip the image
  itself

This has been discussed at the plumbers conference, and there the solution we 
came up with for "does this cam need software whitebalance?" was (AFAIK), check 
if has a V4L2_CID_AUTO_WHITE_BALANCE, if it does not do software whitebalance. 
This of course means we will be doing software whitebalance on things like 
framefrabbers etc. too, so the plan was to combine this with an "is_webcam" 
flag in the capabilities struct. The is_webcam workaround, already shows what 
is wrong with this approach, we are checking for something not being there, 
were we should be checking for the driver asking something actively,

So we need an extensible mechanism to query devices if they could benefit from 
certain additional processing being done on the generated image data.

This sounds a lot like the existing mechanism for v4l2 controls, except that 
these are all read only controls, and not controls which we want to show up in 
v4l control panels like v4l2ucp.

Still I think that using the existing controls mechanism is the best way todo 
this, so therefor I propose to add a number of standard CID's to query the 
things listed above. All these CID's will always be shown by the driver as 
readonly and disabled (as they are not really controls).

Here is an initial proposal for the new CID's, I'm sure the list will grow this 
is just a first revision:

#define V4L2_CTRL_CLASS_CAMERA_PROPERTY 0x009b0000

#define V4L2_CID_CAMERA_PROPERTY_CLASS_BASE \
	(V4L2_CTRL_CLASS_CAMERA_PROPERTY | 0x900)
#define V4L2_CID_CAMERA_PROPERTY_CLASS \
	(V4L2_CTRL_CLASS_CAMERA_PROPERTY | 1)

/* Booleans */
#define V4L2_CID_WANTS_SW_WHITEBALANCE  (V4L2_CID_CAMERA_PROPERTY_CLASS_BASE+1)
#define V4L2_CID_WANTS_SW_AUTO_EXPOSURE (V4L2_CID_CAMERA_PROPERTY_CLASS_BASE+2)
#define V4L2_CID_WANTS_SW_GAMMA_CORRECT (V4L2_CID_CAMERA_PROPERTY_CLASS_BASE+3)
#define V4L2_CID_SENSOR_UPSIDE_DOWN     (V4L2_CID_CAMERA_PROPERTY_CLASS_BASE+4)

/* Fixed point, 16.16 stored in 32 bit integer */
#define V4L2_CID_DEF_GAMMA_CORR_FACTOR  (V4L2_CID_CAMERA_PROPERTY_CLASS_BASE+5)


Please let me know what you think of this proposal, as I would like to move 
forward with this soon.

Regards,

Hans

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: RFC: API to query webcams for various webcam specific properties

[Adam Baker]

On Wednesday 19 November 2008, Hans de Goede wrote:
> Hi All,
>
<snip>
>
> This has been discussed at the plumbers conference, and there the solution
> we came up with for "does this cam need software whitebalance?" was
> (AFAIK), check if has a V4L2_CID_AUTO_WHITE_BALANCE, if it does not do
> software whitebalance. This of course means we will be doing software
> whitebalance on things like framefrabbers etc. too, so the plan was to
> combine this with an "is_webcam" flag in the capabilities struct. The
> is_webcam workaround, already shows what is wrong with this approach, we
> are checking for something not being there, were we should be checking for
> the driver asking something actively,

There also seem to be so many things we might want to control that such an 
inference based system is going to hit other limitations.

>
> So we need an extensible mechanism to query devices if they could benefit
> from certain additional processing being done on the generated image data.
>
> This sounds a lot like the existing mechanism for v4l2 controls, except
> that these are all read only controls, and not controls which we want to
> show up in v4l control panels like v4l2ucp.
>
> Still I think that using the existing controls mechanism is the best way
> todo this, so therefor I propose to add a number of standard CID's to query
> the things listed above. All these CID's will always be shown by the driver
> as readonly and disabled (as they are not really controls).
>

I can see this leading to a lot of drivers having to implement a whole bunch 
of cases in a switch statement to handle these values. Could a simpler 
approach be to have a single ioctl to query the set of controls the driver 
would like to have implemented and the driver then responds with the list of 
tags and default values for the controls it would like implemented.

Someting like:

struct tag
{
u32	tag_id;
u32	tag_value;
};

const tag default_tags[] = { {LIBV4L_CTL_GAMMA, 0x34}, 
{LIBV4L_CTL_LRFLIP,1} };

This could also be a mechanism to address your other RFC as to how to store 
the current settings. The fact that you are already adding code to the kernel 
to provide the list of controls somewhat argues against your point that you 
don't want to add code to the kernel to store the current control settings.
The driver could therefore copy the default control values into somewhere in 
it's device struct to provide a per device instance volatile storage for the 
data.

The reason I prefer in driver storage is that it simplifies the task of 
associating the data with the device. If you have a machine with multiple 
webcams they need to have independent sets of controls per device and you 
shouldn't retain the previous values if the user unplugs one webcam and plugs 
in another that gets the same /dev/videox name.

If you do use shared memory have you considered wheter to use the SysV or 
Posix variant? Both variants provide the required retain while not in use 
functionality but have different naming rules.

Is it necessary to provide a mechanism to notify other libv4l instances that 
the set values have changed? With driver stored values I think it is but if 
you use shared memory they could simply be read each time a frame is 
received.

Adam

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: RFC: API to query webcams for various webcam specific properties

[Hans de Goede]

Adam Baker wrote:
> On Wednesday 19 November 2008, Hans de Goede wrote:
>> Hi All,
>>
> <snip>
>> This has been discussed at the plumbers conference, and there the solution
>> we came up with for "does this cam need software whitebalance?" was
>> (AFAIK), check if has a V4L2_CID_AUTO_WHITE_BALANCE, if it does not do
>> software whitebalance. This of course means we will be doing software
>> whitebalance on things like framefrabbers etc. too, so the plan was to
>> combine this with an "is_webcam" flag in the capabilities struct. The
>> is_webcam workaround, already shows what is wrong with this approach, we
>> are checking for something not being there, were we should be checking for
>> the driver asking something actively,
> 
> There also seem to be so many things we might want to control that such an 
> inference based system is going to hit other limitations.
> 
>> So we need an extensible mechanism to query devices if they could benefit
>> from certain additional processing being done on the generated image data.
>>
>> This sounds a lot like the existing mechanism for v4l2 controls, except
>> that these are all read only controls, and not controls which we want to
>> show up in v4l control panels like v4l2ucp.
>>
>> Still I think that using the existing controls mechanism is the best way
>> todo this, so therefor I propose to add a number of standard CID's to query
>> the things listed above. All these CID's will always be shown by the driver
>> as readonly and disabled (as they are not really controls).
>>
> 
> I can see this leading to a lot of drivers having to implement a whole bunch 
> of cases in a switch statement to handle these values. Could a simpler 
> approach be to have a single ioctl to query the set of controls the driver 
> would like to have implemented and the driver then responds with the list of 
> tags and default values for the controls it would like implemented.
> 
> Someting like:
> 
> struct tag
> {
> u32	tag_id;
> u32	tag_value;
> };
> 
> const tag default_tags[] = { {LIBV4L_CTL_GAMMA, 0x34}, 
> {LIBV4L_CTL_LRFLIP,1} };
> 

Yes, implementing v4l2 controls in drivers can take up quite a bit of code. 
This is a general problem though, not limited to my proposal for new CID's for 
the device/driver to be able to give hints to userspace that certain 
post-processing would be good todo.

We really need to add some framework code to the v4l2 core to make handling 
controls at the driver level easier.

What I envision is at the end having something like you propose above in the 
driver, and then have the v4l2 core do the translating to / from v4l2 control 
calls.

> This could also be a mechanism to address your other RFC as to how to store 
> the current settings. The fact that you are already adding code to the kernel 
> to provide the list of controls somewhat argues against your point that you 
> don't want to add code to the kernel to store the current control settings.
> The driver could therefore copy the default control values into somewhere in 
> it's device struct to provide a per device instance volatile storage for the 
> data.
> 
> The reason I prefer in driver storage is that it simplifies the task of 
> associating the data with the device. If you have a machine with multiple 
> webcams they need to have independent sets of controls per device and you 
> shouldn't retain the previous values if the user unplugs one webcam and plugs 
> in another that gets the same /dev/videox name.
> 
> If you do use shared memory have you considered wheter to use the SysV or 
> Posix variant? Both variants provide the required retain while not in use 
> functionality but have different naming rules.
> 
> Is it necessary to provide a mechanism to notify other libv4l instances that 
> the set values have changed? With driver stored values I think it is but if 
> you use shared memory they could simply be read each time a frame is 
> received.

Yes, the idea is to simply read the values each frame, which is why shared 
memory is prefered as a solution to this. Your worries can be solved by simply 
putting the "card" member of the v4l2_capability struct + the minor no in the 
shared memory segment name / id.

Regards,

Hans

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list