[ANN] IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[ANN] IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Sakari Ailus]

Hi all,

I'd like to announce that we'll have an IRC meeting on #v4l-meeting channel
on the new sensor control interface. The date is next Monday 2012-01-09
14:00 GMT + 2. Most important background information is this; it discusses
how image sensors should be controlled:

<URL:http://www.spinics.net/lists/linux-media/msg40861.html>

These changes currently depend on

- Integer menu controls [1],
- Selection IOCTL for subdevs [2] and
- validate_pipeline() V4L2 subdev pad op.

The full patchset, with all the latest patches for the above, is available
here. What also is there, is the SMIA++ driver which uses these interfaces.
It can be used on the Nokia N9.

<URL:http://www.spinics.net/lists/linux-media/msg41765.html>


Questions and comments are always very, very welcome by e-mail as usual. The
purpose of the meeting is to have a possibility for real-time discussion on
the topic.


[1] http://www.spinics.net/lists/linux-media/msg40796.html

[2] http://www.spinics.net/lists/linux-media/msg41503.html


Kind regards,

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [ANN] IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Sakari Ailus]

Hi,

On Wed, Jan 04, 2012 at 10:56:33AM +0200, Sakari Ailus wrote:
> Hi all,
> 
> I'd like to announce that we'll have an IRC meeting on #v4l-meeting channel
> on the new sensor control interface. The date is next Monday 2012-01-09
> 14:00 GMT + 2. Most important background information is this; it discusses
> how image sensors should be controlled:
> 
> <URL:http://www.spinics.net/lists/linux-media/msg40861.html>
> 
> These changes currently depend on
> 
> - Integer menu controls [1],
> - Selection IOCTL for subdevs [2] and

Based on Laurent's comments to the subdev selection patches, I drew a
diagram to visualise format / crop / scaling and composition configuration.
It's available here:

<URL:http://www.retiisi.org.uk/v4l2/tmp/format.eps>

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

[ANN] Notes on IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Sakari Ailus]

Hi all,


We had an IRC meeting on the new sensor control interface on #v4l-meeting
as scheduled previously. The meeting log is available here:

<URL:http://www.retiisi.org.uk/v4l2/v4l2-sensor-control-interface-2012-01-09.txt>

My notes can be found below.


Accessing V4L2 subdev and MC interfaces in user space: user space libraries
===========================================================================

While the V4L2 subdev and Media controller kernel interface is functionally
comprehensive, it is a relatively low level interface for even for
vendor-specific user space camera libraries. The issue is intensified with
the extension of the pipeline configuration performed using the Media
controller and V4L2 subdev interfaces to cover the image processing
performed on the sensor: this is part of the new sensor control interface.

As we want to encourage SoC vendors to use the V4L2, we need to make this as
easy as possible for them.

The low level camera control libraries can be split into roughly two
categories: those which configure the image pipe and those which deal with
the 3A algorithms. The 3A algorithms are typically proprietary so we
concentrated to the pipeline configuration which is what the Media
controller and V4L2 subdev frameworks have been intended for.

Two libraries already exist for this: libmediactl and libv4l2subdev. The
former deals with topology enumeration and link configuration whereas the
latter is a generic library for V4L2 subdev configuration, including format
configuration.

The new sensor control interface moves the remaining policy decisions to the
user space: how the sensor's image pipe is configured, what pixel rates are
being used on the bus from the sensor to the ISP and how is the blanking
configured.

The role of the new library, called libv4l2pipe, is to interpret text-based
configuration file containing sections for various pipeline format and link
configurations, as well as V4L2 controls: the link frequency is a control as
well; but more on that below. The library may be later on merged to
libv4l2pipeauto which Sakari is working on.

Both pipeline format and link configurations are policy decisions and thus
can be expected to be use case specific. A format configuration is dependent
on a link configuration but the same link configuration can be used with
several format configurations. Thus the two should be defined separately.

A third kind of section will be for setting controls. The only control to be
set will be the link frequency control but a new type of setting warrants a
new section.

A fourth section may be required as well: at this level the frame rate (or
frame time) range makes more sense than the low-level blanking values. The
blanking values can be calculated from the frame time and a flag which tells
whether either horizontal or vertical blanking should be preferred.

A configuration consisting of all the above sections will define the full
pipeline configuration. The library must also provide a way to enumerate,
query and set these configurations.

With the existence of this library and the related new sensor control
interface, the V4L2 supports implementing digital cameras even better than
it used to.

The LGPL 2.1+ license used by libmediactl, libv4l2pipeauto and the future
libv4l2pipe(auto) is not seen an issue for Android to adopt these libraries
either.

In GStreamer middleware, libv4l2pipe is expected to be used by the camera
source component.


The new sensor control interface
================================


The common understanding was that the new sensor control interface is mostly
accepted. No patches have been acked since there have been lots of trivial
and some not so trivial issues in the patchset. There was an exception to
this, which is the pixel_rate field in struct v4l2_mbus_framefmt.

The field is expected to be propagated by the user while the user has no
valid use case to modify it. The agreement was that instead of adding the
field to struct v4l2_mbus_framefmt, a new control will be introduced
instead.

A control has several good properties: it can be implemented where it is
valid: it isn't always possible to accurately specify the pixel rate in some
parts of the pipeline.

Sensor drivers should provide the pixel_rate control in two subdevs: the
pixel array and the one which is opposed to the ISP's bus receiver. The
pixel array's pixel rate is mostly required in the user space whereas the
pixel rate in the bus transmitter subdev (which may have other functionality
as well) is often required by the bus receivers, as well as by the rest of
the ISP.

Ideally the pixel_rate control is related to pads rather than subdevs but 1)
we don't have pad specific controls and 2) we don't stictly need them right
now since there only will be need for a single pixel_rate control per
subdev.

If pixel rate management will be implemented to prevent starting pipelines
which would fail to stream in cases where too high pixel rates are used on
particular subdevs, the concept of pad-specific controls may be later
revisited. Making the pixel_rate control pad-specific only will change the
interface towards the user space if the pad where it is implemented is
non-zero.


-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [ANN] Notes on IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Laurent Pinchart]

Hi Sakari,

On Monday 09 January 2012 18:38:25 Sakari Ailus wrote:
> Hi all,
> 
> We had an IRC meeting on the new sensor control interface on #v4l-meeting
> as scheduled previously. The meeting log is available here:
> 
> <URL:http://www.retiisi.org.uk/v4l2/v4l2-sensor-control-interface-2012-01-0
> 9.txt>
> 
> My notes can be found below.

Thanks for the summary.

> Accessing V4L2 subdev and MC interfaces in user space: user space libraries
> ===========================================================================
> 
> While the V4L2 subdev and Media controller kernel interface is functionally
> comprehensive, it is a relatively low level interface for even for
> vendor-specific user space camera libraries. The issue is intensified with
> the extension of the pipeline configuration performed using the Media
> controller and V4L2 subdev interfaces to cover the image processing
> performed on the sensor: this is part of the new sensor control interface.
> 
> As we want to encourage SoC vendors to use the V4L2, we need to make this
> as easy as possible for them.
> 
> The low level camera control libraries can be split into roughly two
> categories: those which configure the image pipe and those which deal with
> the 3A algorithms. The 3A algorithms are typically proprietary so we
> concentrated to the pipeline configuration which is what the Media
> controller and V4L2 subdev frameworks have been intended for.
> 
> Two libraries already exist for this: libmediactl and libv4l2subdev. The
> former deals with topology enumeration and link configuration whereas the
> latter is a generic library for V4L2 subdev configuration, including format
> configuration.
> 
> The new sensor control interface moves the remaining policy decisions to
> the user space: how the sensor's image pipe is configured, what pixel
> rates are being used on the bus from the sensor to the ISP and how is the
> blanking configured.
> 
> The role of the new library, called libv4l2pipe, is to interpret text-based
> configuration file containing sections for various pipeline format and link
> configurations, as well as V4L2 controls: the link frequency is a control
> as well; but more on that below. The library may be later on merged to
> libv4l2pipeauto which Sakari is working on.
> 
> Both pipeline format and link configurations are policy decisions and thus
> can be expected to be use case specific. A format configuration is
> dependent on a link configuration but the same link configuration can be
> used with several format configurations. Thus the two should be defined
> separately.
> 
> A third kind of section will be for setting controls. The only control to
> be set will be the link frequency control but a new type of setting
> warrants a new section.
> 
> A fourth section may be required as well: at this level the frame rate (or
> frame time) range makes more sense than the low-level blanking values. The
> blanking values can be calculated from the frame time and a flag which
> tells whether either horizontal or vertical blanking should be preferred.

How does one typically select between horizontal and vertical blanking ? Do 
mixed modes make sense ?

> A configuration consisting of all the above sections will define the full
> pipeline configuration. The library must also provide a way to enumerate,
> query and set these configurations.
> 
> With the existence of this library and the related new sensor control
> interface, the V4L2 supports implementing digital cameras even better than
> it used to.
> 
> The LGPL 2.1+ license used by libmediactl, libv4l2pipeauto and the future
> libv4l2pipe(auto) is not seen an issue for Android to adopt these libraries
> either.
> 
> In GStreamer middleware, libv4l2pipe is expected to be used by the camera
> source component.

Should we try to draft how a 3A library should be implemented ? Do you think 
that might have implications on libv4l2pipe ?

> The new sensor control interface
> ================================
> 
> 
> The common understanding was that the new sensor control interface is
> mostly accepted. No patches have been acked since there have been lots of
> trivial and some not so trivial issues in the patchset. There was an
> exception to this, which is the pixel_rate field in struct
> v4l2_mbus_framefmt.
> 
> The field is expected to be propagated by the user while the user has no
> valid use case to modify it. The agreement was that instead of adding the
> field to struct v4l2_mbus_framefmt, a new control will be introduced
> instead.
> 
> A control has several good properties: it can be implemented where it is
> valid: it isn't always possible to accurately specify the pixel rate in
> some parts of the pipeline.
> 
> Sensor drivers should provide the pixel_rate control in two subdevs: the
> pixel array and the one which is opposed to the ISP's bus receiver. The
> pixel array's pixel rate is mostly required in the user space whereas the
> pixel rate in the bus transmitter subdev (which may have other
> functionality as well) is often required by the bus receivers, as well as
> by the rest of the ISP.
> 
> Ideally the pixel_rate control is related to pads rather than subdevs but
> 1) we don't have pad specific controls and 2) we don't stictly need them
> right now since there only will be need for a single pixel_rate control
> per subdev.
> 
> If pixel rate management will be implemented to prevent starting pipelines
> which would fail to stream in cases where too high pixel rates are used on
> particular subdevs, the concept of pad-specific controls may be later
> revisited. Making the pixel_rate control pad-specific only will change the
> interface towards the user space if the pad where it is implemented is
> non-zero.

I'm fine with that. Let's use a control now, we'll revisit this later if 
needed.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Notes on IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Sakari Ailus]

Hi Laurent,

Laurent Pinchart wrote:
> On Monday 09 January 2012 18:38:25 Sakari Ailus wrote:
>> Hi all,
>>
>> We had an IRC meeting on the new sensor control interface on #v4l-meeting
>> as scheduled previously. The meeting log is available here:
>>
>> <URL:http://www.retiisi.org.uk/v4l2/v4l2-sensor-control-interface-2012-01-0
>> 9.txt>
>>
>> My notes can be found below.
>
> Thanks for the summary.
>
>> Accessing V4L2 subdev and MC interfaces in user space: user space libraries
>> ===========================================================================
>>
>> While the V4L2 subdev and Media controller kernel interface is functionally
>> comprehensive, it is a relatively low level interface for even for
>> vendor-specific user space camera libraries. The issue is intensified with
>> the extension of the pipeline configuration performed using the Media
>> controller and V4L2 subdev interfaces to cover the image processing
>> performed on the sensor: this is part of the new sensor control interface.
>>
>> As we want to encourage SoC vendors to use the V4L2, we need to make this
>> as easy as possible for them.
>>
>> The low level camera control libraries can be split into roughly two
>> categories: those which configure the image pipe and those which deal with
>> the 3A algorithms. The 3A algorithms are typically proprietary so we
>> concentrated to the pipeline configuration which is what the Media
>> controller and V4L2 subdev frameworks have been intended for.
>>
>> Two libraries already exist for this: libmediactl and libv4l2subdev. The
>> former deals with topology enumeration and link configuration whereas the
>> latter is a generic library for V4L2 subdev configuration, including format
>> configuration.
>>
>> The new sensor control interface moves the remaining policy decisions to
>> the user space: how the sensor's image pipe is configured, what pixel
>> rates are being used on the bus from the sensor to the ISP and how is the
>> blanking configured.
>>
>> The role of the new library, called libv4l2pipe, is to interpret text-based
>> configuration file containing sections for various pipeline format and link
>> configurations, as well as V4L2 controls: the link frequency is a control
>> as well; but more on that below. The library may be later on merged to
>> libv4l2pipeauto which Sakari is working on.
>>
>> Both pipeline format and link configurations are policy decisions and thus
>> can be expected to be use case specific. A format configuration is
>> dependent on a link configuration but the same link configuration can be
>> used with several format configurations. Thus the two should be defined
>> separately.
>>
>> A third kind of section will be for setting controls. The only control to
>> be set will be the link frequency control but a new type of setting
>> warrants a new section.
>>
>> A fourth section may be required as well: at this level the frame rate (or
>> frame time) range makes more sense than the low-level blanking values. The
>> blanking values can be calculated from the frame time and a flag which
>> tells whether either horizontal or vertical blanking should be preferred.
>
> How does one typically select between horizontal and vertical blanking ? Do
> mixed modes make sense ?

There are minimums and maximums for both. You can increase the frame 
time by increasing value for either or both of them --- to achieve very 
long frame times you may have to use both, but that's not very common in 
practice. I think we should have a flag to tell which one should be 
increased first --- the effect would be to have the minimum possible 
value on the other.

>> A configuration consisting of all the above sections will define the full
>> pipeline configuration. The library must also provide a way to enumerate,
>> query and set these configurations.
>>
>> With the existence of this library and the related new sensor control
>> interface, the V4L2 supports implementing digital cameras even better than
>> it used to.
>>
>> The LGPL 2.1+ license used by libmediactl, libv4l2pipeauto and the future
>> libv4l2pipe(auto) is not seen an issue for Android to adopt these libraries
>> either.
>>
>> In GStreamer middleware, libv4l2pipe is expected to be used by the camera
>> source component.
>
> Should we try to draft how a 3A library should be implemented ? Do you think
> that might have implications on libv4l2pipe ?

We should, yes. I can't see any immediate effects from that to 
libv4l2pipe. libv4l2pipe may need to provide some information to the 3A 
library but that should mostly be it.

>> The new sensor control interface
>> ================================
>>
>>
>> The common understanding was that the new sensor control interface is
>> mostly accepted. No patches have been acked since there have been lots of
>> trivial and some not so trivial issues in the patchset. There was an
>> exception to this, which is the pixel_rate field in struct
>> v4l2_mbus_framefmt.
>>
>> The field is expected to be propagated by the user while the user has no
>> valid use case to modify it. The agreement was that instead of adding the
>> field to struct v4l2_mbus_framefmt, a new control will be introduced
>> instead.
>>
>> A control has several good properties: it can be implemented where it is
>> valid: it isn't always possible to accurately specify the pixel rate in
>> some parts of the pipeline.
>>
>> Sensor drivers should provide the pixel_rate control in two subdevs: the
>> pixel array and the one which is opposed to the ISP's bus receiver. The
>> pixel array's pixel rate is mostly required in the user space whereas the
>> pixel rate in the bus transmitter subdev (which may have other
>> functionality as well) is often required by the bus receivers, as well as
>> by the rest of the ISP.
>>
>> Ideally the pixel_rate control is related to pads rather than subdevs but
>> 1) we don't have pad specific controls and 2) we don't stictly need them
>> right now since there only will be need for a single pixel_rate control
>> per subdev.
>>
>> If pixel rate management will be implemented to prevent starting pipelines
>> which would fail to stream in cases where too high pixel rates are used on
>> particular subdevs, the concept of pad-specific controls may be later
>> revisited. Making the pixel_rate control pad-specific only will change the
>> interface towards the user space if the pad where it is implemented is
>> non-zero.
>
> I'm fine with that. Let's use a control now, we'll revisit this later if
> needed.

Agreed.

-- 
Sakari Ailus
sakari.ailus@iki.fi

Re: [ANN] Notes on IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Laurent Pinchart]

Hi Sakari,

On Monday 09 January 2012 23:32:06 Sakari Ailus wrote:
> Laurent Pinchart wrote:
> > On Monday 09 January 2012 18:38:25 Sakari Ailus wrote:
> >> Hi all,
> >> 
> >> We had an IRC meeting on the new sensor control interface on
> >> #v4l-meeting as scheduled previously. The meeting log is available
> >> here:
> >> 
> >> <URL:http://www.retiisi.org.uk/v4l2/v4l2-sensor-control-interface-2012-0
> >> 1-0 9.txt>
> >> 
> >> My notes can be found below.
> > 
> > Thanks for the summary.
> > 
> >> Accessing V4L2 subdev and MC interfaces in user space: user space
> >> libraries
> >> =======================================================================
> >> ====
> >> 
> >> While the V4L2 subdev and Media controller kernel interface is
> >> functionally comprehensive, it is a relatively low level interface for
> >> even for vendor-specific user space camera libraries. The issue is
> >> intensified with the extension of the pipeline configuration performed
> >> using the Media controller and V4L2 subdev interfaces to cover the
> >> image processing performed on the sensor: this is part of the new
> >> sensor control interface.
> >> 
> >> As we want to encourage SoC vendors to use the V4L2, we need to make
> >> this as easy as possible for them.
> >> 
> >> The low level camera control libraries can be split into roughly two
> >> categories: those which configure the image pipe and those which deal
> >> with the 3A algorithms. The 3A algorithms are typically proprietary so
> >> we concentrated to the pipeline configuration which is what the Media
> >> controller and V4L2 subdev frameworks have been intended for.
> >> 
> >> Two libraries already exist for this: libmediactl and libv4l2subdev. The
> >> former deals with topology enumeration and link configuration whereas
> >> the latter is a generic library for V4L2 subdev configuration,
> >> including format configuration.
> >> 
> >> The new sensor control interface moves the remaining policy decisions to
> >> the user space: how the sensor's image pipe is configured, what pixel
> >> rates are being used on the bus from the sensor to the ISP and how is
> >> the blanking configured.
> >> 
> >> The role of the new library, called libv4l2pipe, is to interpret
> >> text-based configuration file containing sections for various pipeline
> >> format and link configurations, as well as V4L2 controls: the link
> >> frequency is a control as well; but more on that below. The library may
> >> be later on merged to libv4l2pipeauto which Sakari is working on.
> >> 
> >> Both pipeline format and link configurations are policy decisions and
> >> thus can be expected to be use case specific. A format configuration is
> >> dependent on a link configuration but the same link configuration can
> >> be used with several format configurations. Thus the two should be
> >> defined separately.
> >> 
> >> A third kind of section will be for setting controls. The only control
> >> to be set will be the link frequency control but a new type of setting
> >> warrants a new section.
> >> 
> >> A fourth section may be required as well: at this level the frame rate
> >> (or frame time) range makes more sense than the low-level blanking
> >> values. The blanking values can be calculated from the frame time and a
> >> flag which tells whether either horizontal or vertical blanking should
> >> be preferred.
> > 
> > How does one typically select between horizontal and vertical blanking ?
> > Do mixed modes make sense ?
> 
> There are minimums and maximums for both. You can increase the frame
> time by increasing value for either or both of them --- to achieve very
> long frame times you may have to use both, but that's not very common in
> practice. I think we should have a flag to tell which one should be
> increased first --- the effect would be to have the minimum possible
> value on the other.

But how do you decide in practice which one to increase when you're an 
application (or middleware) developer ?

> >> A configuration consisting of all the above sections will define the
> >> full pipeline configuration. The library must also provide a way to
> >> enumerate, query and set these configurations.
> >> 
> >> With the existence of this library and the related new sensor control
> >> interface, the V4L2 supports implementing digital cameras even better
> >> than it used to.
> >> 
> >> The LGPL 2.1+ license used by libmediactl, libv4l2pipeauto and the
> >> future libv4l2pipe(auto) is not seen an issue for Android to adopt
> >> these libraries either.
> >> 
> >> In GStreamer middleware, libv4l2pipe is expected to be used by the
> >> camera source component.
> > 
> > Should we try to draft how a 3A library should be implemented ? Do you
> > think that might have implications on libv4l2pipe ?
> 
> We should, yes. I can't see any immediate effects from that to
> libv4l2pipe. libv4l2pipe may need to provide some information to the 3A
> library but that should mostly be it.
>
> >> The new sensor control interface
> >> ================================
> >> 
> >> 
> >> The common understanding was that the new sensor control interface is
> >> mostly accepted. No patches have been acked since there have been lots
> >> of trivial and some not so trivial issues in the patchset. There was an
> >> exception to this, which is the pixel_rate field in struct
> >> v4l2_mbus_framefmt.
> >> 
> >> The field is expected to be propagated by the user while the user has no
> >> valid use case to modify it. The agreement was that instead of adding
> >> the field to struct v4l2_mbus_framefmt, a new control will be
> >> introduced instead.
> >> 
> >> A control has several good properties: it can be implemented where it is
> >> valid: it isn't always possible to accurately specify the pixel rate in
> >> some parts of the pipeline.
> >> 
> >> Sensor drivers should provide the pixel_rate control in two subdevs: the
> >> pixel array and the one which is opposed to the ISP's bus receiver. The
> >> pixel array's pixel rate is mostly required in the user space whereas
> >> the pixel rate in the bus transmitter subdev (which may have other
> >> functionality as well) is often required by the bus receivers, as well
> >> as by the rest of the ISP.
> >> 
> >> Ideally the pixel_rate control is related to pads rather than subdevs
> >> but 1) we don't have pad specific controls and 2) we don't stictly need
> >> them right now since there only will be need for a single pixel_rate
> >> control per subdev.
> >> 
> >> If pixel rate management will be implemented to prevent starting
> >> pipelines which would fail to stream in cases where too high pixel
> >> rates are used on particular subdevs, the concept of pad-specific
> >> controls may be later revisited. Making the pixel_rate control
> >> pad-specific only will change the interface towards the user space if
> >> the pad where it is implemented is non-zero.
> > 
> > I'm fine with that. Let's use a control now, we'll revisit this later if
> > needed.
> 
> Agreed.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Notes on IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Sakari Ailus]

Hi Laurent,

Laurent Pinchart wrote:
> On Monday 09 January 2012 23:32:06 Sakari Ailus wrote:
>> Laurent Pinchart wrote:
>>> On Monday 09 January 2012 18:38:25 Sakari Ailus wrote:
...
>>>> A fourth section may be required as well: at this level the frame rate
>>>> (or frame time) range makes more sense than the low-level blanking
>>>> values. The blanking values can be calculated from the frame time and a
>>>> flag which tells whether either horizontal or vertical blanking should
>>>> be preferred.
>>>
>>> How does one typically select between horizontal and vertical blanking ?
>>> Do mixed modes make sense ?
>>
>> There are minimums and maximums for both. You can increase the frame
>> time by increasing value for either or both of them --- to achieve very
>> long frame times you may have to use both, but that's not very common in
>> practice. I think we should have a flag to tell which one should be
>> increased first --- the effect would be to have the minimum possible
>> value on the other.
>
> But how do you decide in practice which one to increase when you're an
> application (or middleware) developer ?

I think it's the responsibility of this library to do that, unless the 
user wants really, really precise control in which case they have to 
deal with the blanking values directly. In general it should be the library.

-- 
Sakari Ailus
sakari.ailus@iki.fi

Re: [ANN] Notes on IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Laurent Pinchart]

Hi Sakari,

On Tuesday 10 January 2012 00:26:46 Sakari Ailus wrote:
> Laurent Pinchart wrote:
> > On Monday 09 January 2012 23:32:06 Sakari Ailus wrote:
> >> Laurent Pinchart wrote:
> >>> On Monday 09 January 2012 18:38:25 Sakari Ailus wrote:
> ...
> 
> >>>> A fourth section may be required as well: at this level the frame rate
> >>>> (or frame time) range makes more sense than the low-level blanking
> >>>> values. The blanking values can be calculated from the frame time and
> >>>> a flag which tells whether either horizontal or vertical blanking
> >>>> should be preferred.
> >>> 
> >>> How does one typically select between horizontal and vertical blanking
> >>> ? Do mixed modes make sense ?
> >> 
> >> There are minimums and maximums for both. You can increase the frame
> >> time by increasing value for either or both of them --- to achieve very
> >> long frame times you may have to use both, but that's not very common in
> >> practice. I think we should have a flag to tell which one should be
> >> increased first --- the effect would be to have the minimum possible
> >> value on the other.
> > 
> > But how do you decide in practice which one to increase when you're an
> > application (or middleware) developer ?
> 
> I think it's the responsibility of this library to do that, unless the
> user wants really, really precise control in which case they have to
> deal with the blanking values directly. In general it should be the
> library.

And how does the library decide ? :-)

-- 
Regards,

Laurent Pinchart

Re: [ANN] Notes on IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Sakari Ailus]

Hi Laurent,

Laurent Pinchart wrote:
> On Tuesday 10 January 2012 00:26:46 Sakari Ailus wrote:
>> Laurent Pinchart wrote:
>>> On Monday 09 January 2012 23:32:06 Sakari Ailus wrote:
>>>> Laurent Pinchart wrote:
>>>>> On Monday 09 January 2012 18:38:25 Sakari Ailus wrote:
>> ...
>>
>>>>>> A fourth section may be required as well: at this level the frame rate
>>>>>> (or frame time) range makes more sense than the low-level blanking
>>>>>> values. The blanking values can be calculated from the frame time and
>>>>>> a flag which tells whether either horizontal or vertical blanking
>>>>>> should be preferred.
>>>>>
>>>>> How does one typically select between horizontal and vertical blanking
>>>>> ? Do mixed modes make sense ?
>>>>
>>>> There are minimums and maximums for both. You can increase the frame
>>>> time by increasing value for either or both of them --- to achieve very
>>>> long frame times you may have to use both, but that's not very common in
>>>> practice. I think we should have a flag to tell which one should be
>>>> increased first --- the effect would be to have the minimum possible
>>>> value on the other.
>>>
>>> But how do you decide in practice which one to increase when you're an
>>> application (or middleware) developer ?
>>
>> I think it's the responsibility of this library to do that, unless the
>> user wants really, really precise control in which case they have to
>> deal with the blanking values directly. In general it should be the
>> library.
>
> And how does the library decide ? :-)
>

frame_time = pixel_rate / ((width + hblank) * (height + vblank))

The user gives you frame time and the configuration contains the 
information which one to prefer. Let's say the user prefers hblank (from 
the above):

(width + hblank) * frame_time = pixel_rate / (height + vblank_min)

hblank = pixel_rate / (height + vblank_min) / frame_time - width

width, height, pixel_rate and blankings are as in the pixel array. 
Elsewhere these values may depend on the link frequency and other 
factors so the pixel array is the only reliable place to do this.

-- 
Sakari Ailus
sakari.ailus@iki.fi

Re: [ANN] Notes on IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Laurent Pinchart]

Hi Sakari,

On Tuesday 10 January 2012 10:42:58 Sakari Ailus wrote:
> Laurent Pinchart wrote:
> > On Tuesday 10 January 2012 00:26:46 Sakari Ailus wrote:
> >> Laurent Pinchart wrote:
> >>> On Monday 09 January 2012 23:32:06 Sakari Ailus wrote:
> >>>> Laurent Pinchart wrote:
> >>>>> On Monday 09 January 2012 18:38:25 Sakari Ailus wrote:
> >> ...
> >> 
> >>>>>> A fourth section may be required as well: at this level the frame
> >>>>>> rate (or frame time) range makes more sense than the low-level
> >>>>>> blanking values. The blanking values can be calculated from the
> >>>>>> frame time and a flag which tells whether either horizontal or
> >>>>>> vertical blanking should be preferred.
> >>>>> 
> >>>>> How does one typically select between horizontal and vertical
> >>>>> blanking ? Do mixed modes make sense ?
> >>>> 
> >>>> There are minimums and maximums for both. You can increase the frame
> >>>> time by increasing value for either or both of them --- to achieve
> >>>> very long frame times you may have to use both, but that's not very
> >>>> common in practice. I think we should have a flag to tell which one
> >>>> should be increased first --- the effect would be to have the minimum
> >>>> possible value on the other.
> >>> 
> >>> But how do you decide in practice which one to increase when you're an
> >>> application (or middleware) developer ?
> >> 
> >> I think it's the responsibility of this library to do that, unless the
> >> user wants really, really precise control in which case they have to
> >> deal with the blanking values directly. In general it should be the
> >> library.
> > 
> > And how does the library decide ? :-)
> 
> frame_time = pixel_rate / ((width + hblank) * (height + vblank))
> 
> The user gives you frame time and the configuration contains the
> information which one to prefer. Let's say the user prefers hblank (from
> the above):

That was my question, how does the user decide whether hblank or vblank is 
preferred ?

> (width + hblank) * frame_time = pixel_rate / (height + vblank_min)
> 
> hblank = pixel_rate / (height + vblank_min) / frame_time - width
> 
> width, height, pixel_rate and blankings are as in the pixel array.
> Elsewhere these values may depend on the link frequency and other
> factors so the pixel array is the only reliable place to do this.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Notes on IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Sakari Ailus]

Laurent Pinchart wrote:
> Hi Sakari,
>
> On Tuesday 10 January 2012 10:42:58 Sakari Ailus wrote:
>> Laurent Pinchart wrote:
>>> On Tuesday 10 January 2012 00:26:46 Sakari Ailus wrote:
>>>> Laurent Pinchart wrote:
>>>>> On Monday 09 January 2012 23:32:06 Sakari Ailus wrote:
>>>>>> Laurent Pinchart wrote:
>>>>>>> On Monday 09 January 2012 18:38:25 Sakari Ailus wrote:
>>>> ...
>>>>
>>>>>>>> A fourth section may be required as well: at this level the frame
>>>>>>>> rate (or frame time) range makes more sense than the low-level
>>>>>>>> blanking values. The blanking values can be calculated from the
>>>>>>>> frame time and a flag which tells whether either horizontal or
>>>>>>>> vertical blanking should be preferred.
>>>>>>>
>>>>>>> How does one typically select between horizontal and vertical
>>>>>>> blanking ? Do mixed modes make sense ?
>>>>>>
>>>>>> There are minimums and maximums for both. You can increase the frame
>>>>>> time by increasing value for either or both of them --- to achieve
>>>>>> very long frame times you may have to use both, but that's not very
>>>>>> common in practice. I think we should have a flag to tell which one
>>>>>> should be increased first --- the effect would be to have the minimum
>>>>>> possible value on the other.
>>>>>
>>>>> But how do you decide in practice which one to increase when you're an
>>>>> application (or middleware) developer ?
>>>>
>>>> I think it's the responsibility of this library to do that, unless the
>>>> user wants really, really precise control in which case they have to
>>>> deal with the blanking values directly. In general it should be the
>>>> library.
>>>
>>> And how does the library decide ? :-)
>>
>> frame_time = pixel_rate / ((width + hblank) * (height + vblank))
>>
>> The user gives you frame time and the configuration contains the
>> information which one to prefer. Let's say the user prefers hblank (from
>> the above):
>
> That was my question, how does the user decide whether hblank or vblank is
> preferred ?

I think that should be defined in the configuration itself. It's very 
unlikely there's any need to change this dynamically.

-- 
Sakari Ailus
sakari.ailus@iki.fi

Re: [ANN] Notes on IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Laurent Pinchart]

Hi Sakari,

On Tuesday 10 January 2012 10:52:26 Sakari Ailus wrote:
> Laurent Pinchart wrote:
> > On Tuesday 10 January 2012 10:42:58 Sakari Ailus wrote:
> >> Laurent Pinchart wrote:
> >>> On Tuesday 10 January 2012 00:26:46 Sakari Ailus wrote:
> >>>> Laurent Pinchart wrote:
> >>>>> On Monday 09 January 2012 23:32:06 Sakari Ailus wrote:
> >>>>>> Laurent Pinchart wrote:
> >>>>>>> On Monday 09 January 2012 18:38:25 Sakari Ailus wrote:
> >>>> ...
> >>>> 
> >>>>>>>> A fourth section may be required as well: at this level the frame
> >>>>>>>> rate (or frame time) range makes more sense than the low-level
> >>>>>>>> blanking values. The blanking values can be calculated from the
> >>>>>>>> frame time and a flag which tells whether either horizontal or
> >>>>>>>> vertical blanking should be preferred.
> >>>>>>> 
> >>>>>>> How does one typically select between horizontal and vertical
> >>>>>>> blanking ? Do mixed modes make sense ?
> >>>>>> 
> >>>>>> There are minimums and maximums for both. You can increase the frame
> >>>>>> time by increasing value for either or both of them --- to achieve
> >>>>>> very long frame times you may have to use both, but that's not very
> >>>>>> common in practice. I think we should have a flag to tell which one
> >>>>>> should be increased first --- the effect would be to have the
> >>>>>> minimum possible value on the other.
> >>>>> 
> >>>>> But how do you decide in practice which one to increase when you're
> >>>>> an application (or middleware) developer ?
> >>>> 
> >>>> I think it's the responsibility of this library to do that, unless the
> >>>> user wants really, really precise control in which case they have to
> >>>> deal with the blanking values directly. In general it should be the
> >>>> library.
> >>> 
> >>> And how does the library decide ? :-)
> >> 
> >> frame_time = pixel_rate / ((width + hblank) * (height + vblank))
> >> 
> >> The user gives you frame time and the configuration contains the
> >> information which one to prefer. Let's say the user prefers hblank (from
> > 
> >> the above):
> > That was my question, how does the user decide whether hblank or vblank
> > is preferred ?
> 
> I think that should be defined in the configuration itself. It's very
> unlikely there's any need to change this dynamically.

Sure, but that's not my point. How does the user decide in the first place 
when writing the configuration ?

-- 
Regards,

Laurent Pinchart

Re: [ANN] Notes on IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Sakari Ailus]

Hi Laurent,

Laurent Pinchart wrote:
> On Tuesday 10 January 2012 10:52:26 Sakari Ailus wrote:
>> Laurent Pinchart wrote:
>>> On Tuesday 10 January 2012 10:42:58 Sakari Ailus wrote:
>>>> Laurent Pinchart wrote:
>>>>> On Tuesday 10 January 2012 00:26:46 Sakari Ailus wrote:
>>>>>> Laurent Pinchart wrote:
>>>>>>> On Monday 09 January 2012 23:32:06 Sakari Ailus wrote:
>>>>>>>> Laurent Pinchart wrote:
>>>>>>>>> On Monday 09 January 2012 18:38:25 Sakari Ailus wrote:
>>>>>> ...
>>>>>>
>>>>>>>>>> A fourth section may be required as well: at this level the frame
>>>>>>>>>> rate (or frame time) range makes more sense than the low-level
>>>>>>>>>> blanking values. The blanking values can be calculated from the
>>>>>>>>>> frame time and a flag which tells whether either horizontal or
>>>>>>>>>> vertical blanking should be preferred.
>>>>>>>>>
>>>>>>>>> How does one typically select between horizontal and vertical
>>>>>>>>> blanking ? Do mixed modes make sense ?
>>>>>>>>
>>>>>>>> There are minimums and maximums for both. You can increase the frame
>>>>>>>> time by increasing value for either or both of them --- to achieve
>>>>>>>> very long frame times you may have to use both, but that's not very
>>>>>>>> common in practice. I think we should have a flag to tell which one
>>>>>>>> should be increased first --- the effect would be to have the
>>>>>>>> minimum possible value on the other.
>>>>>>>
>>>>>>> But how do you decide in practice which one to increase when you're
>>>>>>> an application (or middleware) developer ?
>>>>>>
>>>>>> I think it's the responsibility of this library to do that, unless the
>>>>>> user wants really, really precise control in which case they have to
>>>>>> deal with the blanking values directly. In general it should be the
>>>>>> library.
>>>>>
>>>>> And how does the library decide ? :-)
>>>>
>>>> frame_time = pixel_rate / ((width + hblank) * (height + vblank))
>>>>
>>>> The user gives you frame time and the configuration contains the
>>>> information which one to prefer. Let's say the user prefers hblank (from
>>>
>>>> the above):
>>> That was my question, how does the user decide whether hblank or vblank
>>> is preferred ?
>>
>> I think that should be defined in the configuration itself. It's very
>> unlikely there's any need to change this dynamically.
>
> Sure, but that's not my point. How does the user decide in the first place
> when writing the configuration ?

That's a policy decision. There may be multiple reasons for that, but 
the most obvious I can think of is a decision between amortised memory 
access rate and the rolling shutter effect. There may be other factors 
like time of year, phase of moon and colour of the neighbour's cat to 
name a few. :-)

It might also make sense to be able to specify a minimum for horizontal 
blanking which is bigger than the minimum the sensor allows. But that's 
going to the fine details.

-- 
Sakari Ailus
sakari.ailus@iki.fi

Re: [ANN] Notes on IRC meeting on new sensor control interface, 2012-01-09 14:00 GMT+2

[Tuukka Toivonen]

On Tuesday 10 January 2012 12:05:48 Laurent Pinchart wrote:
> > > That was my question, how does the user decide whether hblank or vblank
> > > is preferred ?
> > 
> > I think that should be defined in the configuration itself. It's very
> > unlikely there's any need to change this dynamically.
> 
> Sure, but that's not my point. How does the user decide in the first place
> when writing the configuration ?

As Sakari finally told,
- More vertical blanking
  -> less rolling shutter effect (ie. less slanted image) and more time to do
      some processing between frames if wanted or needed
- More horizontal blanking -> smoother data rate, especially if there are
  buffers which can contain just couple of lines

I would say that generally vertical blanking is preferred unless there are
small buffers which allow required decrease in data rate for later stages in
image processing pipeline with larger horizontal blanking.

- Tuukka
---------------------------------------------------------------------
Intel Finland Oy
Registered Address: PL 281, 00181 Helsinki 
Business Identity Code: 0357606 - 4 
Domiciled in Helsinki 

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.