From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mail-ee0-f46.google.com ([74.125.83.46]:44283 "EHLO mail-ee0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753111Ab1LaOm6 (ORCPT ); Sat, 31 Dec 2011 09:42:58 -0500 Received: by eekc4 with SMTP id c4so14322879eek.19 for ; Sat, 31 Dec 2011 06:42:57 -0800 (PST) Message-ID: <4EFF1F6B.2090009@gmail.com> Date: Sat, 31 Dec 2011 15:42:51 +0100 From: Sylwester Nawrocki MIME-Version: 1.0 To: Sakari Ailus CC: linux-media@vger.kernel.org, laurent.pinchart@ideasonboard.com, t.stanislaws@samsung.com, dacohen@gmail.com, andriy.shevchenko@linux.intel.com, g.liakhovetski@gmx.de, hverkuil@xs4all.nl Subject: Re: [RFC 2/3] v4l: Image source control class References: <20111201143044.GI29805@valkosipuli.localdomain> <1323876147-18107-2-git-send-email-sakari.ailus@iki.fi> In-Reply-To: <1323876147-18107-2-git-send-email-sakari.ailus@iki.fi> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Sender: linux-media-owner@vger.kernel.org List-ID: Hi Sakari, thanks for the patch. On 12/14/2011 04:22 PM, Sakari Ailus wrote: > Add image source control class. This control class is intended to contain > low level controls which deal with control of the image capture process --- > the A/D converter in image sensors, for example. > > Signed-off-by: Sakari Ailus > --- > Documentation/DocBook/media/v4l/controls.xml | 101 ++++++++++++++++++++ > .../DocBook/media/v4l/vidioc-g-ext-ctrls.xml | 6 + > drivers/media/video/v4l2-ctrls.c | 10 ++ > include/linux/videodev2.h | 10 ++ > 4 files changed, 127 insertions(+), 0 deletions(-) > > diff --git a/Documentation/DocBook/media/v4l/controls.xml b/Documentation/DocBook/media/v4l/controls.xml > index 3bc5ee8..69ede83 100644 > --- a/Documentation/DocBook/media/v4l/controls.xml > +++ b/Documentation/DocBook/media/v4l/controls.xml > @@ -3356,6 +3356,107 @@ interface and may change in the future. > > > > + > +
> + Image Source Control Reference > + > + > + Experimental > + > + This is an + linkend="experimental">experimental interface and may > + change in the future. > + > + > + > + The Image Source control class is intended for low-level > + control of image source devices such as image sensors. The > + devices feature an analogue to digital converter and a bus > + transmitter to transmit the image data out of the device. > + > + > + > + Image Source Control IDs > + > + > + > + > + > + > + > + > + > + > + ID > + Type > + Description > + > + > + > + > + > + V4L2_CID_IMAGE_SOURCE_CLASS > + class > + > + > + The IMAGE_SOURCE class descriptor. > + > + > + V4L2_CID_IMAGE_SOURCE_VBLANK > + integer > + > + > + Vertical blanking. The idle > + preriod after every frame during which no image data is s/preriod/period > + produced. The unit of vertical blanking is a line. Every > + line has length of the image width plus horizontal > + blanking at the pixel clock specified by struct > + v4l2_mbus_framefmt + />. > + > + > + V4L2_CID_IMAGE_SOURCE_HBLANK > + integer > + > + > + Horizontal blanking. The idle > + preriod after every line of image data during which no s/preriod/period > + image data is produced. The unit of horizontal blanking is > + pixels. > + > + > + V4L2_CID_IMAGE_SOURCE_LINK_FREQ > + integer menu > + > + > + Image source's data bus frequency. > + Together with the media bus pixel code, bus type (clock > + cycles per sample), the data bus frequency defines the > + pixel clock. The > + frame rate can be calculated from the pixel clock, image > + width and height and horizontal and vertical blanking. The > + frame rate control is performed by selecting the desired > + horizontal and vertical blanking. > + > + > + > + V4L2_CID_IMAGE_SOURCE_ANALOGUE_GAIN > + integer > + > + > + Analogue gain is gain affecting > + all colour components in the pixel matrix. The gain > + operation is performed in the analogue domain before A/D > + conversion. > + > + > + > + > + > +
> + > +
> + > > > 8 [1]. http://patchwork.linuxtv.org/patch/8923/ -- Thanks, Sylwester From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mail-ee0-f46.google.com ([74.125.83.46]:57882 "EHLO mail-ee0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753111Ab1LaOxS (ORCPT ); Sat, 31 Dec 2011 09:53:18 -0500 Received: by eekc4 with SMTP id c4so14327026eek.19 for ; Sat, 31 Dec 2011 06:53:17 -0800 (PST) Message-ID: <4EFF21D5.3080500@gmail.com> Date: Sat, 31 Dec 2011 15:53:09 +0100 From: Sylwester Nawrocki MIME-Version: 1.0 To: Sakari Ailus CC: linux-media@vger.kernel.org, laurent.pinchart@ideasonboard.com, t.stanislaws@samsung.com, dacohen@gmail.com, andriy.shevchenko@linux.intel.com, g.liakhovetski@gmx.de, hverkuil@xs4all.nl Subject: Re: [RFC 2/3] v4l: Image source control class References: <20111201143044.GI29805@valkosipuli.localdomain> <1323876147-18107-2-git-send-email-sakari.ailus@iki.fi> <4EFF1F6B.2090009@gmail.com> In-Reply-To: <4EFF1F6B.2090009@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Sender: linux-media-owner@vger.kernel.org List-ID: On 12/31/2011 03:42 PM, Sylwester Nawrocki wrote: >> >> + case V4L2_CID_IMAGE_SOURCE_CLASS: return "Image source controls"; >> + case V4L2_CID_IMAGE_SOURCE_VBLANK: return "Vertical blanking"; > > nit: have you considered making it "Blanking, horizontal" Oops, it supposed to be: "Blanking, vertical" >> + case V4L2_CID_IMAGE_SOURCE_HBLANK: return "Horizontal blanking"; > > and "Blanking, vertical" ? and "Blanking, horizontal" :/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from smtp-68.nebula.fi ([83.145.220.68]:37626 "EHLO smtp-68.nebula.fi" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754340Ab1LAOau (ORCPT ); Thu, 1 Dec 2011 09:30:50 -0500 Date: Thu, 1 Dec 2011 16:30:44 +0200 From: Sakari Ailus To: linux-media@vger.kernel.org Cc: laurent.pinchart@ideasonboard.com, hverkuil@xs4all.nl, snjw23@gmail.com, g.liakhovetski@gmx.de, dacohen@gmail.com, hdegoede@redhat.com, pawel@osciak.com Subject: [RFC] On controlling sensors Message-ID: <20111201143044.GI29805@valkosipuli.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Sender: linux-media-owner@vger.kernel.org List-ID: Hi all, I've been working a few years with digital cameras and Linux and come to the conclusion that the way the sensors are typically configured through V4L2 subdevs is far from optimal for the use of the low level software that typically accesses it, nor we make efficient use of the standard V4L2 subdev API currently. The regular S_FMT and S_PARM IOCTLs are fine as a high level interface. This RFC does _not_ take a negative stance against them nor suggest they should ever be deprecated. No new interfaces are proposed for regular V4L2 applications either. Comments to this RFC are the most welcome. What's wrong ============ Pixel array, binning and scaling -------------------------------- ENUM_FRAMESIZES interface makes no sense on V4L2 subdevice user space interface. Why so? Since the size of the sensor's pixel array is constant. Other sizes are derived from that by scaling and cropping. These days most sensors can perform a lot of image processing by themselves --- for example binning and cropping are available on most sensors the pixel array of which is larger than 640x480 or so. More advanced sensors can also perform scaling. We have existing interface in V4L2 subdev to configure scaling. The sensors should impelement this interface. This may involve exporting more than one subdevs from the sensor driver. Frame rates ----------- Sensors are utterly unaware of the whole concept of frame rate. Instead, they deal with pixel clocks, horizontal and vertical blanking, binning and scaling factors and link frequencies. The same goes for the low level software controlling automatic white balance and exposure on the sensor: this information is required to set and get detailed timing information from the sensor. However, the current control mechanisms provided by the V4L2 subdev interface provides neither. It is also impossible to provide detailed frame rate information using the current interfaces. Pixel rate --------- The pixel rate is important as well. The maximum pixel rate for some of the hardware blocks may be lower than those earlier in the pipeline. This becomes an issue in the OMAP 3 ISP where only the CSI-2 receiver can handle pixel rates up to 200 Mp/s whereas the rest of the ISP blocks only can do 100 Mp/s. The current APIs do not reveal this, but it is also impossible to know the output data rate of the sensor in order to come up with a valid configuration. How to fix this =============== Sensors should be controlled by exposing their natural parameters to the user. This means the following: Link frequency as control ------------------------- Typically the hardware allows few different link frequencies to be used. The actual data rate may depend on the bits-per-pixel value and how many lanes are being used on serial links. Togethet with the binning and scaling factors, the link pixel rate defines how fast the sensor pixel array can be read. The pixel rate at the pixel array is important for calculating the sensor timing for the user of the control algorithms. Link frequency is chosen over pixel rate since pixel rate is dependent on the format of the data --- bits per pixel. Horizontal and vertical blanking as controls -------------------------------------------- Horizontal and vertical blanking typically is relatively freely configurable in a range between a lower and an upper limit. The limits are hardware specific. The unit of the horizontal blanking is line and for vertical blanking it's pixel. Together with the link rate, vertical and horizontal blanking, image width and height define the frame rate. There are also other reasons why the user is interested in the blanking information, such as to minimise the rolling shutter effect or amortise the data rate on the memory bus. Binning, scaling and cropping configuration ------------------------------------------- Binning, scaling and cropping must be configured using the same Media controller and V4L2 subdevice APIs as the image pipe inside the ISPs. It makes really no difference whether the scaler is located in the ISP or in the sensor: the API for configuring it must be the same. This may mean exposing the sensor as multiple subdevices to make it configurable using the V4L2 subdevice interface. A sensor could look like this, for example: pixel array:0 [crop] ---> 0:binner:1 ---> 0:ISP's CSI2 receiver The pixel array's pixel rate would only be defined after configuring the binner and the link rate between the sensor and the ISP. The reason for this is that the pixel clock in the pixel array may well be higher than after the scaler. But the pixel clock, and the limits for vertical and horizontal blanking are only available after the binning factor and the link rate have been specified. The blanking must be thus specified after configuring the formats on the pipeline. Proposal ======== In the spec and the interfaces not much would change: only the purposes we are currently using the interfaces would change a little bit. The sensor drivers would need to make more use of the V4L2 subdev interface. Control class for low-level controls ------------------------------------ A new control class will be created for low level controls. It is to be called V4L2_CTRL_CLASS_LL, where "LL" is for "low level". Better proposals on the name of the class are welcome. My feeling is that we might get lots of controls to this class over time. Controls in this class will include V4L2_CID_LL_VBLANK, V4L2_CID_LL_HBLANK and V4L2_CID_LL_LINK_FREQ. Link frequency is chosen instead of pixel rate since the pixel rate is dependent on the format. That would make the available values depend on selected format which I consider potentially confusing. The same class should likely hold the controls for separate analogue and digital gains and per-component gains. Per-component gains are typically digital-only, but some sensors support per-component analog gains. So we should have common gains and per-component gains, both digital and analog. Pixel rate as part of struct v4l2_mbus_pixelfmt ----------------------------------------------- Pixel rate would be added to the struct v4l2_mbus_pixelfmt in kP/s (kilopixels per second). struct v4l2_mbus_framefmt { __u32 width; __u32 height; __u32 code; __u32 field; __u32 colorspace; __u32 pixel_rate; __u32 reserved[6]; }; Sensors with only a few pre-defined configurations -------------------------------------------------- Some sensors, such as the infamous Omnivision ones, only provide register lists to configure the sensor from a few pre-defined configurations. These sensor drivers cannot meaningfully expose the low-level API for controlling them and are bound to continue to provide high level information on supported resolutions and frame rates. The generic user space should have no issues in recognising such sensors from the ones which do provide the low-level controls. We still need to define the exact method to distinguish them. I don't like very much that we would have different ways to configure devices, but as we simply do not have enough information to configure these properly I do not really see a way around it. Regular V4L2 applications ------------------------- Controlling sensors this way is obviously the most suitable for embedded systems. Desktops might not care. But still on embedded systems one may well want to run regular V4L2 applications. The solution for this is the same as previously agreed on: a pipeline setup library, a configuration configuration program and a libv4l2 plugin using the library. Vendors are encouraged to provide their own libraries which know the features of their hardware the best and can best configure them as well. It also would require using these libraries on devices which have not needed them before: if you combine a sensor driver exposing this kind of an interface with a regular camera bridge which just exposes regular V4L2 API, it won't work as such. To avoid creating double effort in pipeline configuration, the bridge driver also must expose V4L2 subdev API so the full pipeline configuration can be done from the user space. Cheers, -- Sakari Ailus e-mail: sakari.ailus@iki.fi jabber/XMPP/Gmail: sailus@retiisi.org.uk From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mail-ee0-f46.google.com ([74.125.83.46]:49368 "EHLO mail-ee0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756377Ab1LOWTp (ORCPT ); Thu, 15 Dec 2011 17:19:45 -0500 Received: by eekc4 with SMTP id c4so2616097eek.19 for ; Thu, 15 Dec 2011 14:19:44 -0800 (PST) Message-ID: <4EEA727C.6010906@gmail.com> Date: Thu, 15 Dec 2011 23:19:40 +0100 From: Sylwester Nawrocki MIME-Version: 1.0 To: Sakari Ailus CC: linux-media@vger.kernel.org, laurent.pinchart@ideasonboard.com, t.stanislaws@samsung.com, dacohen@gmail.com, andriy.shevchenko@linux.intel.com, g.liakhovetski@gmx.de, hverkuil@xs4all.nl Subject: Re: [RFC 1/3] v4l: Add pixel clock to struct v4l2_mbus_framefmt References: <20111201143044.GI29805@valkosipuli.localdomain> <1323876147-18107-1-git-send-email-sakari.ailus@iki.fi> <4EEA6AAE.80405@gmail.com> <20111215220150.GH3677@valkosipuli.localdomain> In-Reply-To: <20111215220150.GH3677@valkosipuli.localdomain> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Sender: linux-media-owner@vger.kernel.org List-ID: On 12/15/2011 11:01 PM, Sakari Ailus wrote: >>> __u32 >>> - reserved[7] >>> + pixel_clock >>> + Pixel clock in kHz. This clock is the maximum rate at >>> + which pixels are transferred on the bus. The pixel_clock >>> + field is read-only. >> >> I searched a couple of datasheets to find out where I could use this pixel_clock >> field but didn't find any so far. I haven't tried too hard though ;) >> There seems to be more benefits from having the link frequency control. > > There are a few reasons to have the pixel clock available to the user space. > > The previously existing reason is that the user may get information on the > pixel rates, including cases where the pixel rate of a subdev isn't enough > for the streaming to be possible. Earlier on it just failed. Such cases are > common on the OMAP 3 ISP, for example. > > The second reason is to provide that for timing calculations in the user > space. Fair enough. Perhaps, if I have worked more with image signal processing algorithms in user space I would not ask about that in the first place :-) > >> It might be easy to confuse pixel_clock with the bus clock. The bus clock is >> often referred in datasheets as Pixel Clock (PCLK, AFAIU it's described with >> link frequency in your RFC). IMHO your original proposal was better, i.e. >> using more explicit pixel_rate. Also why it is in kHz ? Doesn't it make more >> sense to use bits or pixels per second ? > > Oh, yes, now that you mention it I did call it pixel rate. I'm fine > withrenaming it back to e.g. "pixelrate". I'm fine with that too, sounds good! > > I picked kHz since the 32-bit field would allow rates up to 4 GiP/s. Not > sure if that's overkill though. Could be. But in practice it should give > good enough precision this way, too. All right, however I was more concerned by the "Hz" part, rather than "k" ;) It might be good to have the relevant unit defined in the spec, to avoid misinterpretation and future interoperability issues . >>> + >>> + >>> + __u32 >>> + reserved[6] >>> Reserved for future extensions. Applications and drivers must >>> set the array to zero. >>> >>> diff --git a/include/linux/v4l2-mediabus.h b/include/linux/v4l2-mediabus.h >>> index 5ea7f75..76a0df2 100644 >>> --- a/include/linux/v4l2-mediabus.h >>> +++ b/include/linux/v4l2-mediabus.h >>> @@ -101,6 +101,7 @@ enum v4l2_mbus_pixelcode { >>> * @code: data format code (from enum v4l2_mbus_pixelcode) >>> * @field: used interlacing type (from enum v4l2_field) >>> * @colorspace: colorspace of the data (from enum v4l2_colorspace) >>> + * @pixel_clock: pixel clock, in kHz >>> */ >>> struct v4l2_mbus_framefmt { >>> __u32 width; >>> @@ -108,7 +109,8 @@ struct v4l2_mbus_framefmt { >>> __u32 code; >>> __u32 field; >>> __u32 colorspace; >>> - __u32 reserved[7]; >>> + __u32 pixel_clock; >> >> I'm wondering, whether it is worth to make it 'pixelclock' for consistency >> with other fields? Perhaps it would make more sense to have color_space and >> pixel_clock. > > "pixelrate" is fine for me. Ack. -- Regards, Sylwester From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from smtp.nokia.com ([147.243.1.48]:27120 "EHLO mgw-sa02.nokia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757248Ab1LNPW6 (ORCPT ); Wed, 14 Dec 2011 10:22:58 -0500 From: Sakari Ailus To: linux-media@vger.kernel.org Cc: laurent.pinchart@ideasonboard.com, snjw23@gmail.com, t.stanislaws@samsung.com, dacohen@gmail.com, andriy.shevchenko@linux.intel.com, g.liakhovetski@gmx.de, hverkuil@xs4all.nl Subject: [RFC 2/3] v4l: Image source control class Date: Wed, 14 Dec 2011 17:22:26 +0200 Message-Id: <1323876147-18107-2-git-send-email-sakari.ailus@iki.fi> In-Reply-To: <20111201143044.GI29805@valkosipuli.localdomain> References: <20111201143044.GI29805@valkosipuli.localdomain> Sender: linux-media-owner@vger.kernel.org List-ID: Add image source control class. This control class is intended to contain low level controls which deal with control of the image capture process --- the A/D converter in image sensors, for example. Signed-off-by: Sakari Ailus --- Documentation/DocBook/media/v4l/controls.xml | 101 ++++++++++++++++++++ .../DocBook/media/v4l/vidioc-g-ext-ctrls.xml | 6 + drivers/media/video/v4l2-ctrls.c | 10 ++ include/linux/videodev2.h | 10 ++ 4 files changed, 127 insertions(+), 0 deletions(-) diff --git a/Documentation/DocBook/media/v4l/controls.xml b/Documentation/DocBook/media/v4l/controls.xml index 3bc5ee8..69ede83 100644 --- a/Documentation/DocBook/media/v4l/controls.xml +++ b/Documentation/DocBook/media/v4l/controls.xml @@ -3356,6 +3356,107 @@ interface and may change in the future. + +
+ Image Source Control Reference + + + Experimental + + This is an experimental interface and may + change in the future. + + + + The Image Source control class is intended for low-level + control of image source devices such as image sensors. The + devices feature an analogue to digital converter and a bus + transmitter to transmit the image data out of the device. + + + + Image Source Control IDs + + + + + + + + + + + ID + Type + Description + + + + + + V4L2_CID_IMAGE_SOURCE_CLASS + class + + + The IMAGE_SOURCE class descriptor. + + + V4L2_CID_IMAGE_SOURCE_VBLANK + integer + + + Vertical blanking. The idle + preriod after every frame during which no image data is + produced. The unit of vertical blanking is a line. Every + line has length of the image width plus horizontal + blanking at the pixel clock specified by struct + v4l2_mbus_framefmt . + + + V4L2_CID_IMAGE_SOURCE_HBLANK + integer + + + Horizontal blanking. The idle + preriod after every line of image data during which no + image data is produced. The unit of horizontal blanking is + pixels. + + + V4L2_CID_IMAGE_SOURCE_LINK_FREQ + integer menu + + + Image source's data bus frequency. + Together with the media bus pixel code, bus type (clock + cycles per sample), the data bus frequency defines the + pixel clock. The + frame rate can be calculated from the pixel clock, image + width and height and horizontal and vertical blanking. The + frame rate control is performed by selecting the desired + horizontal and vertical blanking. + + + + V4L2_CID_IMAGE_SOURCE_ANALOGUE_GAIN + integer + + + Analogue gain is gain affecting + all colour components in the pixel matrix. The gain + operation is performed in the analogue domain before A/D + conversion. + + + + + +
+ +
+