* [RFC v3 0/9] Sub-device configuration models
@ 2024-11-29 9:51 Sakari Ailus
2024-11-29 9:51 ` [RFC v3 1/9] media: Documentation: Rework embedded data documentation Sakari Ailus
` (9 more replies)
0 siblings, 10 replies; 25+ messages in thread
From: Sakari Ailus @ 2024-11-29 9:51 UTC (permalink / raw)
To: linux-media
Cc: hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hello everyone,
I've been recently working (with others) on sub-device streams support as
well as on internal pads. The two can be used to make sub-device
configuration more versatile.
At the same time, the added interfaces are much more useful if we require
specific semantics of those interfaces, so that the user space knows
exactly what e.g. a given selection target signifies. However, as the same
selection rectangle could be used for a different purpose on a non-raw
sensor device, we need a way to tell how should the user space determine
how to use a given interface.
I'm proposing to solve this problem by introducing sub-device
configuration models, and by the common raw sensor model, also present in
this patchset, in particular.
This has been (and will, for some time, continue to be) the reason why I
have reviewed few sensor driver related patches lately. As we're
introducing a new interface, it's beneficial to be able to use that
interface right from the start, rather than trying to later on offer
compatibility support, which is almost always a fair amount of work with
less than desirable results in the driver.
With this solved, I believe we can enable the use of the streams UAPI.
Comments are welcome.
RFC v2 (labelled as PATCH) is
<20241122100633.8971-1-sakari.ailus@linux.intel.com> on LMML.
RFC v1 is <20241011075535.588140-1-sakari.ailus@linux.intel.com> on LMML.
We may end up postponing merging three last patches (those adding
controls) until current control values are moved to sub-device (and video
device) state. Discussion is needed on the topic.
The compiled documentation can be found here
<URL:https://www.retiisi.eu/~sailus/v4l2/tmp/common-raw/output/userspace-api/media/v4l/dev-subdev.html#media-subdev-config-model-common-raw-sensor>.
since v2:
- Remove the CROP_BOUNDS rectangle on the image data sink pad. It's
redundant.
- Remove scaling from the sink compose rectangle, leaving binning and
sub-sampling.
- Introduce scaling as well as post-scaler digital crop on source pad
(image stream). This is a new patch, preceding control changes.
- Use monospace font for selection targets.
- Improvements in referring to previous configuration steps in text
description.
since v1:
- Rework the documentation according to the comments (spelling fixes,
alignment mostly).
- Include separate patches for binning and sub-sampling controls.
- Allow binning, scaling and scaling to be configured using the compose
rectangle. This will change to just scaling in the three last patches.
Sakari Ailus (9):
media: Documentation: Rework embedded data documentation
media: Documentation: Add a hyphen to list-based
media: Documentation: Reword split of sensor driver to two classes
media: Documentation: Add subdev configuration models, raw sensor
model
media: Documentation: Add scaling and post-scaler crop for common raw
media: uapi: Add V4L2_CID_CONFIG_MODEL control
media: uapi: Add V4L2_CID_BINNING control for binning configuration
media: uapi: Add controls for sub-sampling configuration
media: Documentation: Add binning and sub-sampling controls
.../media/drivers/camera-sensor.rst | 53 ++-
.../media/v4l/common-raw-sensor.dia | 441 ++++++++++++++++++
.../media/v4l/common-raw-sensor.svg | 134 ++++++
.../userspace-api/media/v4l/dev-subdev.rst | 2 +
.../media/v4l/ext-ctrls-camera.rst | 40 ++
.../media/v4l/ext-ctrls-image-process.rst | 4 +
.../media/v4l/subdev-config-model.rst | 228 +++++++++
drivers/media/v4l2-core/v4l2-ctrls-defs.c | 9 +
include/uapi/linux/v4l2-controls.h | 6 +
9 files changed, 903 insertions(+), 14 deletions(-)
create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.dia
create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.svg
create mode 100644 Documentation/userspace-api/media/v4l/subdev-config-model.rst
base-commit: f188c7293d34f9ac50c1005da2613d519d97db83
--
2.39.5
^ permalink raw reply [flat|nested] 25+ messages in thread
* [RFC v3 1/9] media: Documentation: Rework embedded data documentation
2024-11-29 9:51 [RFC v3 0/9] Sub-device configuration models Sakari Ailus
@ 2024-11-29 9:51 ` Sakari Ailus
2024-11-29 9:51 ` [RFC v3 2/9] media: Documentation: Add a hyphen to list-based Sakari Ailus
` (8 subsequent siblings)
9 siblings, 0 replies; 25+ messages in thread
From: Sakari Ailus @ 2024-11-29 9:51 UTC (permalink / raw)
To: linux-media
Cc: hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Rework embedded data documentation by removing the reference to the pixel
data stream. The specific documentation of the embedded data interface
will be elsewhere, in the near future either CCS or common raw camera
sensor model documentation.
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
.../userspace-api/media/drivers/camera-sensor.rst | 13 ++++++-------
1 file changed, 6 insertions(+), 7 deletions(-)
diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
index dc415b8f6c8e..8e1083417ae1 100644
--- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
+++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
@@ -111,13 +111,12 @@ the sensor configuration for the captured frame back to the host. While CSI-2 is
the most common data interface used by such sensors, embedded data can be
available on other interfaces as well.
-Such sensors expose two internal sink pads (pads that have both the
-``MEDIA_PAD_FL_SINK <MEDIA-PAD-FL-SINK>`` and ``MEDIA_PAD_FL_INTERNAL
-<MEDIA-PAD-FL-INTERNAL>`` flags set) to model the source of the image and
-embedded data streams. Both of these pads produces a single stream, and the
-sub-device routes those streams to the external (source) pad. If the sub-device
-driver supports disabling embedded data, this can be done by disabling the
-embedded data route via the ``VIDIOC_SUBDEV_S_ROUTING`` IOCTL.
+Embedded data support is indicated by the precence of an internal sink pad (pad
+that has both the ``MEDIA_PAD_FL_SINK <MEDIA-PAD-FL-SINK>`` and
+``MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>`` flags set) with a metadata
+format to model the embedded data stream. If the sub-device driver supports
+disabling embedded data, this can be done by disabling the embedded data route
+via the ``VIDIOC_SUBDEV_S_ROUTING`` IOCTL.
In general, changing the embedded data format from the driver-configured values
is not supported. The height of the metadata is device-specific and the width
--
2.39.5
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [RFC v3 2/9] media: Documentation: Add a hyphen to list-based
2024-11-29 9:51 [RFC v3 0/9] Sub-device configuration models Sakari Ailus
2024-11-29 9:51 ` [RFC v3 1/9] media: Documentation: Rework embedded data documentation Sakari Ailus
@ 2024-11-29 9:51 ` Sakari Ailus
2024-12-05 8:04 ` Jacopo Mondi
2024-11-29 9:51 ` [RFC v3 3/9] media: Documentation: Reword split of sensor driver to two classes Sakari Ailus
` (7 subsequent siblings)
9 siblings, 1 reply; 25+ messages in thread
From: Sakari Ailus @ 2024-11-29 9:51 UTC (permalink / raw)
To: linux-media
Cc: hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Add a hyphen to list-based for uniform spelling in camera-sensor.rst.
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
Documentation/userspace-api/media/drivers/camera-sensor.rst | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
index 8e1083417ae1..9a1e8aa9fc77 100644
--- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
+++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
@@ -26,10 +26,10 @@ of cropping and scaling operations from the device's pixel array's size.
An example of such a driver is the CCS driver.
-Register list based drivers
+Register list-based drivers
~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Register list based drivers generally, instead of able to configure the device
+Register list-based drivers generally, instead of able to configure the device
they control based on user requests, are limited to a number of preset
configurations that combine a number of different parameters that on hardware
level are independent. How a driver picks such configuration is based on the
@@ -67,7 +67,7 @@ is pixels and the unit of the ``V4L2_CID_VBLANK`` is lines. The pixel rate in
the sensor's **pixel array** is specified by ``V4L2_CID_PIXEL_RATE`` in the same
sub-device. The unit of that control is pixels per second.
-Register list based drivers need to implement read-only sub-device nodes for the
+Register list-based drivers need to implement read-only sub-device nodes for the
purpose. Devices that are not register list based need these to configure the
device's internal processing pipeline.
--
2.39.5
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [RFC v3 3/9] media: Documentation: Reword split of sensor driver to two classes
2024-11-29 9:51 [RFC v3 0/9] Sub-device configuration models Sakari Ailus
2024-11-29 9:51 ` [RFC v3 1/9] media: Documentation: Rework embedded data documentation Sakari Ailus
2024-11-29 9:51 ` [RFC v3 2/9] media: Documentation: Add a hyphen to list-based Sakari Ailus
@ 2024-11-29 9:51 ` Sakari Ailus
2024-12-05 8:06 ` Jacopo Mondi
2024-11-29 9:51 ` [RFC v3 4/9] media: Documentation: Add subdev configuration models, raw sensor model Sakari Ailus
` (6 subsequent siblings)
9 siblings, 1 reply; 25+ messages in thread
From: Sakari Ailus @ 2024-11-29 9:51 UTC (permalink / raw)
To: linux-media
Cc: hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
The sensor drivers do not configure the output size of the sensors but the
entire internal pipeline. Reflect this in the documentation.
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
.../userspace-api/media/drivers/camera-sensor.rst | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
index 9a1e8aa9fc77..bc55c861fb69 100644
--- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
+++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
@@ -10,11 +10,13 @@ used to control the camera sensor drivers.
You may also find :ref:`media_writing_camera_sensor_drivers` useful.
-Frame size
-----------
+Sensor internal pipeline configuration
+--------------------------------------
-There are two distinct ways to configure the frame size produced by camera
-sensors.
+Camera sensors have an internal processing pipeline including cropping and
+binning functionality. The sensor drivers belong to two distinct classes, freely
+configurable and register list-based drivers, depending on how the driver
+configures this functionality.
Freely configurable camera sensor drivers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--
2.39.5
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [RFC v3 4/9] media: Documentation: Add subdev configuration models, raw sensor model
2024-11-29 9:51 [RFC v3 0/9] Sub-device configuration models Sakari Ailus
` (2 preceding siblings ...)
2024-11-29 9:51 ` [RFC v3 3/9] media: Documentation: Reword split of sensor driver to two classes Sakari Ailus
@ 2024-11-29 9:51 ` Sakari Ailus
2024-12-04 12:10 ` Jacopo Mondi
2024-11-29 9:51 ` [RFC v3 5/9] media: Documentation: Add scaling and post-scaler crop for common raw Sakari Ailus
` (5 subsequent siblings)
9 siblings, 1 reply; 25+ messages in thread
From: Sakari Ailus @ 2024-11-29 9:51 UTC (permalink / raw)
To: linux-media
Cc: hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Sub-device configuration models define what V4L2 API elements are
available on a compliant sub-device and how do they behave.
The patch also adds a model for common raw sensors.
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
.../media/drivers/camera-sensor.rst | 4 +
.../media/v4l/common-raw-sensor.dia | 441 ++++++++++++++++++
.../media/v4l/common-raw-sensor.svg | 134 ++++++
.../userspace-api/media/v4l/dev-subdev.rst | 2 +
.../media/v4l/subdev-config-model.rst | 208 +++++++++
5 files changed, 789 insertions(+)
create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.dia
create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.svg
create mode 100644 Documentation/userspace-api/media/v4l/subdev-config-model.rst
diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
index bc55c861fb69..5bc4c79d230c 100644
--- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
+++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
@@ -18,6 +18,8 @@ binning functionality. The sensor drivers belong to two distinct classes, freely
configurable and register list-based drivers, depending on how the driver
configures this functionality.
+Also see :ref:`media_subdev_config_model_common_raw_sensor`.
+
Freely configurable camera sensor drivers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -105,6 +107,8 @@ values programmed by the register sequences. The default values of these
controls shall be 0 (disabled). Especially these controls shall not be inverted,
independently of the sensor's mounting rotation.
+.. _media_using_camera_sensor_drivers_embedded_data:
+
Embedded data
-------------
diff --git a/Documentation/userspace-api/media/v4l/common-raw-sensor.dia b/Documentation/userspace-api/media/v4l/common-raw-sensor.dia
new file mode 100644
index 000000000000..aa927527eae3
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/common-raw-sensor.dia
@@ -0,0 +1,441 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<dia:diagram xmlns:dia="http://www.lysator.liu.se/~alla/dia/">
+ <dia:diagramdata>
+ <dia:attribute name="background">
+ <dia:color val="#ffffffff"/>
+ </dia:attribute>
+ <dia:attribute name="pagebreak">
+ <dia:color val="#000099ff"/>
+ </dia:attribute>
+ <dia:attribute name="paper">
+ <dia:composite type="paper">
+ <dia:attribute name="name">
+ <dia:string>#A4#</dia:string>
+ </dia:attribute>
+ <dia:attribute name="tmargin">
+ <dia:real val="2.8222"/>
+ </dia:attribute>
+ <dia:attribute name="bmargin">
+ <dia:real val="2.8222"/>
+ </dia:attribute>
+ <dia:attribute name="lmargin">
+ <dia:real val="2.8222"/>
+ </dia:attribute>
+ <dia:attribute name="rmargin">
+ <dia:real val="2.8222"/>
+ </dia:attribute>
+ <dia:attribute name="is_portrait">
+ <dia:boolean val="true"/>
+ </dia:attribute>
+ <dia:attribute name="scaling">
+ <dia:real val="1"/>
+ </dia:attribute>
+ <dia:attribute name="fitto">
+ <dia:boolean val="false"/>
+ </dia:attribute>
+ </dia:composite>
+ </dia:attribute>
+ <dia:attribute name="grid">
+ <dia:composite type="grid">
+ <dia:attribute name="dynamic">
+ <dia:boolean val="true"/>
+ </dia:attribute>
+ <dia:attribute name="width_x">
+ <dia:real val="1"/>
+ </dia:attribute>
+ <dia:attribute name="width_y">
+ <dia:real val="1"/>
+ </dia:attribute>
+ <dia:attribute name="visible_x">
+ <dia:int val="1"/>
+ </dia:attribute>
+ <dia:attribute name="visible_y">
+ <dia:int val="1"/>
+ </dia:attribute>
+ <dia:composite type="color"/>
+ </dia:composite>
+ </dia:attribute>
+ <dia:attribute name="color">
+ <dia:color val="#d8e5e5ff"/>
+ </dia:attribute>
+ <dia:attribute name="guides"/>
+ <dia:attribute name="guide_color">
+ <dia:color val="#00ff00ff"/>
+ </dia:attribute>
+ <dia:attribute name="display">
+ <dia:composite type="display">
+ <dia:attribute name="antialiased">
+ <dia:boolean val="true"/>
+ </dia:attribute>
+ <dia:attribute name="snap-to-grid">
+ <dia:boolean val="false"/>
+ </dia:attribute>
+ <dia:attribute name="snap-to-guides">
+ <dia:boolean val="true"/>
+ </dia:attribute>
+ <dia:attribute name="snap-to-object">
+ <dia:boolean val="true"/>
+ </dia:attribute>
+ <dia:attribute name="show-grid">
+ <dia:boolean val="true"/>
+ </dia:attribute>
+ <dia:attribute name="show-guides">
+ <dia:boolean val="true"/>
+ </dia:attribute>
+ <dia:attribute name="show-connection-points">
+ <dia:boolean val="true"/>
+ </dia:attribute>
+ </dia:composite>
+ </dia:attribute>
+ </dia:diagramdata>
+ <dia:layer name="Background" visible="true" connectable="false"/>
+ <dia:layer name="Background" visible="true" connectable="false"/>
+ <dia:layer name="Background" visible="true" connectable="true" active="true">
+ <dia:object type="Standard - Box" version="0" id="O0">
+ <dia:attribute name="obj_pos">
+ <dia:point val="16.35,11.5"/>
+ </dia:attribute>
+ <dia:attribute name="obj_bb">
+ <dia:rectangle val="16.3,11.45;25.8,18.45"/>
+ </dia:attribute>
+ <dia:attribute name="elem_corner">
+ <dia:point val="16.35,11.5"/>
+ </dia:attribute>
+ <dia:attribute name="elem_width">
+ <dia:real val="9.4000000000000021"/>
+ </dia:attribute>
+ <dia:attribute name="elem_height">
+ <dia:real val="6.9000000000000021"/>
+ </dia:attribute>
+ <dia:attribute name="show_background">
+ <dia:boolean val="true"/>
+ </dia:attribute>
+ </dia:object>
+ <dia:object type="Geometric - Perfect Circle" version="1" id="O1">
+ <dia:attribute name="obj_pos">
+ <dia:point val="25.5627,14.578"/>
+ </dia:attribute>
+ <dia:attribute name="obj_bb">
+ <dia:rectangle val="25.4627,14.478;26.0191,15.0344"/>
+ </dia:attribute>
+ <dia:attribute name="meta">
+ <dia:composite type="dict"/>
+ </dia:attribute>
+ <dia:attribute name="elem_corner">
+ <dia:point val="25.5627,14.578"/>
+ </dia:attribute>
+ <dia:attribute name="elem_width">
+ <dia:real val="0.35638032780853468"/>
+ </dia:attribute>
+ <dia:attribute name="elem_height">
+ <dia:real val="0.35638032780853468"/>
+ </dia:attribute>
+ <dia:attribute name="line_width">
+ <dia:real val="0.10000000000000001"/>
+ </dia:attribute>
+ <dia:attribute name="line_colour">
+ <dia:color val="#000000ff"/>
+ </dia:attribute>
+ <dia:attribute name="fill_colour">
+ <dia:color val="#ffffffff"/>
+ </dia:attribute>
+ <dia:attribute name="show_background">
+ <dia:boolean val="true"/>
+ </dia:attribute>
+ <dia:attribute name="line_style">
+ <dia:enum val="0"/>
+ <dia:real val="1"/>
+ </dia:attribute>
+ <dia:attribute name="flip_horizontal">
+ <dia:boolean val="false"/>
+ </dia:attribute>
+ <dia:attribute name="flip_vertical">
+ <dia:boolean val="false"/>
+ </dia:attribute>
+ <dia:attribute name="subscale">
+ <dia:real val="1"/>
+ </dia:attribute>
+ </dia:object>
+ <dia:object type="Geometric - Perfect Circle" version="1" id="O2">
+ <dia:attribute name="obj_pos">
+ <dia:point val="17.7702,13.2656"/>
+ </dia:attribute>
+ <dia:attribute name="obj_bb">
+ <dia:rectangle val="17.6702,13.1656;18.24,13.7354"/>
+ </dia:attribute>
+ <dia:attribute name="meta">
+ <dia:composite type="dict"/>
+ </dia:attribute>
+ <dia:attribute name="elem_corner">
+ <dia:point val="17.7702,13.2656"/>
+ </dia:attribute>
+ <dia:attribute name="elem_width">
+ <dia:real val="0.36980024191863681"/>
+ </dia:attribute>
+ <dia:attribute name="elem_height">
+ <dia:real val="0.36980024191863681"/>
+ </dia:attribute>
+ <dia:attribute name="line_width">
+ <dia:real val="0.10000000000000001"/>
+ </dia:attribute>
+ <dia:attribute name="line_colour">
+ <dia:color val="#000000ff"/>
+ </dia:attribute>
+ <dia:attribute name="fill_colour">
+ <dia:color val="#ffffffff"/>
+ </dia:attribute>
+ <dia:attribute name="show_background">
+ <dia:boolean val="true"/>
+ </dia:attribute>
+ <dia:attribute name="line_style">
+ <dia:enum val="0"/>
+ <dia:real val="1"/>
+ </dia:attribute>
+ <dia:attribute name="flip_horizontal">
+ <dia:boolean val="false"/>
+ </dia:attribute>
+ <dia:attribute name="flip_vertical">
+ <dia:boolean val="false"/>
+ </dia:attribute>
+ <dia:attribute name="subscale">
+ <dia:real val="1"/>
+ </dia:attribute>
+ </dia:object>
+ <dia:object type="Geometric - Perfect Circle" version="1" id="O3">
+ <dia:attribute name="obj_pos">
+ <dia:point val="17.7464,16.2056"/>
+ </dia:attribute>
+ <dia:attribute name="obj_bb">
+ <dia:rectangle val="17.6464,16.1056;18.2162,16.6754"/>
+ </dia:attribute>
+ <dia:attribute name="meta">
+ <dia:composite type="dict"/>
+ </dia:attribute>
+ <dia:attribute name="elem_corner">
+ <dia:point val="17.7464,16.2056"/>
+ </dia:attribute>
+ <dia:attribute name="elem_width">
+ <dia:real val="0.36980024191863681"/>
+ </dia:attribute>
+ <dia:attribute name="elem_height">
+ <dia:real val="0.36980024191863681"/>
+ </dia:attribute>
+ <dia:attribute name="line_width">
+ <dia:real val="0.10000000000000001"/>
+ </dia:attribute>
+ <dia:attribute name="line_colour">
+ <dia:color val="#000000ff"/>
+ </dia:attribute>
+ <dia:attribute name="fill_colour">
+ <dia:color val="#ffffffff"/>
+ </dia:attribute>
+ <dia:attribute name="show_background">
+ <dia:boolean val="true"/>
+ </dia:attribute>
+ <dia:attribute name="line_style">
+ <dia:enum val="0"/>
+ <dia:real val="1"/>
+ </dia:attribute>
+ <dia:attribute name="flip_horizontal">
+ <dia:boolean val="false"/>
+ </dia:attribute>
+ <dia:attribute name="flip_vertical">
+ <dia:boolean val="false"/>
+ </dia:attribute>
+ <dia:attribute name="subscale">
+ <dia:real val="1"/>
+ </dia:attribute>
+ </dia:object>
+ <dia:object type="Standard - Line" version="0" id="O4">
+ <dia:attribute name="obj_pos">
+ <dia:point val="18.1609,16.3413"/>
+ </dia:attribute>
+ <dia:attribute name="obj_bb">
+ <dia:rectangle val="18.1016,14.5712;25.6221,16.4007"/>
+ </dia:attribute>
+ <dia:attribute name="conn_endpoints">
+ <dia:point val="18.1609,16.3413"/>
+ <dia:point val="25.5627,14.7562"/>
+ </dia:attribute>
+ <dia:attribute name="numcp">
+ <dia:int val="1"/>
+ </dia:attribute>
+ <dia:attribute name="line_style">
+ <dia:enum val="1"/>
+ </dia:attribute>
+ <dia:attribute name="end_arrow">
+ <dia:enum val="1"/>
+ </dia:attribute>
+ <dia:attribute name="end_arrow_length">
+ <dia:real val="0.5"/>
+ </dia:attribute>
+ <dia:attribute name="end_arrow_width">
+ <dia:real val="0.5"/>
+ </dia:attribute>
+ <dia:connections>
+ <dia:connection handle="0" to="O3" connection="8"/>
+ <dia:connection handle="1" to="O1" connection="2"/>
+ </dia:connections>
+ </dia:object>
+ <dia:object type="Standard - Line" version="0" id="O5">
+ <dia:attribute name="obj_pos">
+ <dia:point val="18.14,13.4505"/>
+ </dia:attribute>
+ <dia:attribute name="obj_bb">
+ <dia:rectangle val="18.0821,13.3926;25.6206,14.9674"/>
+ </dia:attribute>
+ <dia:attribute name="conn_endpoints">
+ <dia:point val="18.14,13.4505"/>
+ <dia:point val="25.5627,14.7562"/>
+ </dia:attribute>
+ <dia:attribute name="numcp">
+ <dia:int val="1"/>
+ </dia:attribute>
+ <dia:attribute name="end_arrow">
+ <dia:enum val="1"/>
+ </dia:attribute>
+ <dia:attribute name="end_arrow_length">
+ <dia:real val="0.5"/>
+ </dia:attribute>
+ <dia:attribute name="end_arrow_width">
+ <dia:real val="0.5"/>
+ </dia:attribute>
+ <dia:connections>
+ <dia:connection handle="0" to="O2" connection="3"/>
+ <dia:connection handle="1" to="O1" connection="2"/>
+ </dia:connections>
+ </dia:object>
+ <dia:object type="Standard - Line" version="0" id="O6">
+ <dia:attribute name="obj_pos">
+ <dia:point val="25.919,14.7562"/>
+ </dia:attribute>
+ <dia:attribute name="obj_bb">
+ <dia:rectangle val="25.8686,14.3879;31.0068,15.0497"/>
+ </dia:attribute>
+ <dia:attribute name="conn_endpoints">
+ <dia:point val="25.919,14.7562"/>
+ <dia:point val="30.9564,14.7131"/>
+ </dia:attribute>
+ <dia:attribute name="numcp">
+ <dia:int val="1"/>
+ </dia:attribute>
+ <dia:attribute name="end_arrow">
+ <dia:enum val="1"/>
+ </dia:attribute>
+ <dia:attribute name="end_arrow_length">
+ <dia:real val="0.5"/>
+ </dia:attribute>
+ <dia:attribute name="end_arrow_width">
+ <dia:real val="0.5"/>
+ </dia:attribute>
+ <dia:connections>
+ <dia:connection handle="0" to="O1" connection="3"/>
+ </dia:connections>
+ </dia:object>
+ <dia:object type="Standard - Text" version="1" id="O7">
+ <dia:attribute name="obj_pos">
+ <dia:point val="17.9551,13.2656"/>
+ </dia:attribute>
+ <dia:attribute name="obj_bb">
+ <dia:rectangle val="17.9551,12.5181;22.7051,13.2656"/>
+ </dia:attribute>
+ <dia:attribute name="text">
+ <dia:composite type="text">
+ <dia:attribute name="string">
+ <dia:string>#image data (1)#</dia:string>
+ </dia:attribute>
+ <dia:attribute name="font">
+ <dia:font family="sans" style="0" name="Helvetica"/>
+ </dia:attribute>
+ <dia:attribute name="height">
+ <dia:real val="0.80000000000000004"/>
+ </dia:attribute>
+ <dia:attribute name="pos">
+ <dia:point val="17.9551,13.1131"/>
+ </dia:attribute>
+ <dia:attribute name="color">
+ <dia:color val="#000000ff"/>
+ </dia:attribute>
+ <dia:attribute name="alignment">
+ <dia:enum val="0"/>
+ </dia:attribute>
+ </dia:composite>
+ </dia:attribute>
+ <dia:attribute name="valign">
+ <dia:enum val="1"/>
+ </dia:attribute>
+ <dia:connections>
+ <dia:connection handle="0" to="O2" connection="1"/>
+ </dia:connections>
+ </dia:object>
+ <dia:object type="Standard - Text" version="1" id="O8">
+ <dia:attribute name="obj_pos">
+ <dia:point val="17.9313,16.5754"/>
+ </dia:attribute>
+ <dia:attribute name="obj_bb">
+ <dia:rectangle val="17.9313,16.5754;24.1238,17.3229"/>
+ </dia:attribute>
+ <dia:attribute name="text">
+ <dia:composite type="text">
+ <dia:attribute name="string">
+ <dia:string>#embedded data (2)#</dia:string>
+ </dia:attribute>
+ <dia:attribute name="font">
+ <dia:font family="sans" style="0" name="Helvetica"/>
+ </dia:attribute>
+ <dia:attribute name="height">
+ <dia:real val="0.80000000000000004"/>
+ </dia:attribute>
+ <dia:attribute name="pos">
+ <dia:point val="17.9313,17.1704"/>
+ </dia:attribute>
+ <dia:attribute name="color">
+ <dia:color val="#000000ff"/>
+ </dia:attribute>
+ <dia:attribute name="alignment">
+ <dia:enum val="0"/>
+ </dia:attribute>
+ </dia:composite>
+ </dia:attribute>
+ <dia:attribute name="valign">
+ <dia:enum val="0"/>
+ </dia:attribute>
+ <dia:connections>
+ <dia:connection handle="0" to="O3" connection="0"/>
+ </dia:connections>
+ </dia:object>
+ <dia:object type="Standard - Text" version="1" id="O9">
+ <dia:attribute name="obj_pos">
+ <dia:point val="26.1,14.125"/>
+ </dia:attribute>
+ <dia:attribute name="obj_bb">
+ <dia:rectangle val="26.1,13.53;30.7475,14.2775"/>
+ </dia:attribute>
+ <dia:attribute name="text">
+ <dia:composite type="text">
+ <dia:attribute name="string">
+ <dia:string>#source pad (0)#</dia:string>
+ </dia:attribute>
+ <dia:attribute name="font">
+ <dia:font family="sans" style="0" name="Helvetica"/>
+ </dia:attribute>
+ <dia:attribute name="height">
+ <dia:real val="0.80000000000000004"/>
+ </dia:attribute>
+ <dia:attribute name="pos">
+ <dia:point val="26.1,14.125"/>
+ </dia:attribute>
+ <dia:attribute name="color">
+ <dia:color val="#000000ff"/>
+ </dia:attribute>
+ <dia:attribute name="alignment">
+ <dia:enum val="0"/>
+ </dia:attribute>
+ </dia:composite>
+ </dia:attribute>
+ <dia:attribute name="valign">
+ <dia:enum val="3"/>
+ </dia:attribute>
+ </dia:object>
+ </dia:layer>
+</dia:diagram>
diff --git a/Documentation/userspace-api/media/v4l/common-raw-sensor.svg b/Documentation/userspace-api/media/v4l/common-raw-sensor.svg
new file mode 100644
index 000000000000..1d6055da2519
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/common-raw-sensor.svg
@@ -0,0 +1,134 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="296pt" height="142pt" viewBox="0 0 296 142" version="1.1">
+<defs>
+<g>
+<symbol overflow="visible" id="glyph0-0">
+<path style="stroke:none;" d="M 0.640625 2.265625 L 0.640625 -9.015625 L 7.03125 -9.015625 L 7.03125 2.265625 Z M 1.359375 1.546875 L 6.328125 1.546875 L 6.328125 -8.296875 L 1.359375 -8.296875 Z M 1.359375 1.546875 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-1">
+<path style="stroke:none;" d="M 1.203125 -7 L 2.359375 -7 L 2.359375 0 L 1.203125 0 Z M 1.203125 -9.71875 L 2.359375 -9.71875 L 2.359375 -8.265625 L 1.203125 -8.265625 Z M 1.203125 -9.71875 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-2">
+<path style="stroke:none;" d="M 6.65625 -5.65625 C 6.9375 -6.164062 7.273438 -6.546875 7.671875 -6.796875 C 8.078125 -7.046875 8.550781 -7.171875 9.09375 -7.171875 C 9.820312 -7.171875 10.382812 -6.914062 10.78125 -6.40625 C 11.175781 -5.894531 11.375 -5.164062 11.375 -4.21875 L 11.375 0 L 10.21875 0 L 10.21875 -4.1875 C 10.21875 -4.851562 10.097656 -5.347656 9.859375 -5.671875 C 9.628906 -6.003906 9.269531 -6.171875 8.78125 -6.171875 C 8.1875 -6.171875 7.710938 -5.972656 7.359375 -5.578125 C 7.015625 -5.179688 6.84375 -4.640625 6.84375 -3.953125 L 6.84375 0 L 5.6875 0 L 5.6875 -4.1875 C 5.6875 -4.863281 5.566406 -5.363281 5.328125 -5.6875 C 5.097656 -6.007812 4.734375 -6.171875 4.234375 -6.171875 C 3.648438 -6.171875 3.179688 -5.96875 2.828125 -5.5625 C 2.484375 -5.164062 2.3125 -4.628906 2.3125 -3.953125 L 2.3125 0 L 1.15625 0 L 1.15625 -7 L 2.3125 -7 L 2.3125 -5.90625 C 2.582031 -6.332031 2.898438 -6.648438 3.265625 -6.859375 C 3.628906 -7.066406 4.0625 -7.171875 4.5625 -7.171875 C 5.070312 -7.171875 5.503906 -7.039062 5.859375 -6.78125 C 6.222656 -6.519531 6.488281 -6.144531 6.65625 -5.65625 Z M 6.65625 -5.65625 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-3">
+<path style="stroke:none;" d="M 4.390625 -3.515625 C 3.460938 -3.515625 2.816406 -3.40625 2.453125 -3.1875 C 2.097656 -2.976562 1.921875 -2.617188 1.921875 -2.109375 C 1.921875 -1.703125 2.050781 -1.378906 2.3125 -1.140625 C 2.582031 -0.898438 2.953125 -0.78125 3.421875 -0.78125 C 4.054688 -0.78125 4.566406 -1.003906 4.953125 -1.453125 C 5.335938 -1.910156 5.53125 -2.515625 5.53125 -3.265625 L 5.53125 -3.515625 Z M 6.671875 -4 L 6.671875 0 L 5.53125 0 L 5.53125 -1.0625 C 5.269531 -0.632812 4.941406 -0.316406 4.546875 -0.109375 C 4.160156 0.0859375 3.679688 0.1875 3.109375 0.1875 C 2.390625 0.1875 1.816406 -0.015625 1.390625 -0.421875 C 0.972656 -0.828125 0.765625 -1.363281 0.765625 -2.03125 C 0.765625 -2.820312 1.023438 -3.414062 1.546875 -3.8125 C 2.078125 -4.21875 2.867188 -4.421875 3.921875 -4.421875 L 5.53125 -4.421875 L 5.53125 -4.53125 C 5.53125 -5.0625 5.351562 -5.46875 5 -5.75 C 4.65625 -6.039062 4.171875 -6.1875 3.546875 -6.1875 C 3.140625 -6.1875 2.75 -6.140625 2.375 -6.046875 C 2 -5.953125 1.632812 -5.8125 1.28125 -5.625 L 1.28125 -6.671875 C 1.695312 -6.835938 2.101562 -6.960938 2.5 -7.046875 C 2.894531 -7.128906 3.28125 -7.171875 3.65625 -7.171875 C 4.675781 -7.171875 5.429688 -6.90625 5.921875 -6.375 C 6.421875 -5.851562 6.671875 -5.0625 6.671875 -4 Z M 6.671875 -4 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-4">
+<path style="stroke:none;" d="M 5.8125 -3.578125 C 5.8125 -4.410156 5.640625 -5.054688 5.296875 -5.515625 C 4.953125 -5.972656 4.46875 -6.203125 3.84375 -6.203125 C 3.226562 -6.203125 2.75 -5.972656 2.40625 -5.515625 C 2.0625 -5.054688 1.890625 -4.410156 1.890625 -3.578125 C 1.890625 -2.753906 2.0625 -2.113281 2.40625 -1.65625 C 2.75 -1.195312 3.226562 -0.96875 3.84375 -0.96875 C 4.46875 -0.96875 4.953125 -1.195312 5.296875 -1.65625 C 5.640625 -2.113281 5.8125 -2.753906 5.8125 -3.578125 Z M 6.953125 -0.875 C 6.953125 0.320312 6.6875 1.207031 6.15625 1.78125 C 5.632812 2.363281 4.828125 2.65625 3.734375 2.65625 C 3.328125 2.65625 2.945312 2.625 2.59375 2.5625 C 2.238281 2.507812 1.890625 2.421875 1.546875 2.296875 L 1.546875 1.171875 C 1.890625 1.359375 2.222656 1.492188 2.546875 1.578125 C 2.878906 1.671875 3.21875 1.71875 3.5625 1.71875 C 4.3125 1.71875 4.875 1.519531 5.25 1.125 C 5.625 0.726562 5.8125 0.132812 5.8125 -0.65625 L 5.8125 -1.234375 C 5.570312 -0.816406 5.265625 -0.503906 4.890625 -0.296875 C 4.523438 -0.0976562 4.082031 0 3.5625 0 C 2.707031 0 2.015625 -0.328125 1.484375 -0.984375 C 0.960938 -1.640625 0.703125 -2.503906 0.703125 -3.578125 C 0.703125 -4.660156 0.960938 -5.53125 1.484375 -6.1875 C 2.015625 -6.84375 2.707031 -7.171875 3.5625 -7.171875 C 4.082031 -7.171875 4.523438 -7.066406 4.890625 -6.859375 C 5.265625 -6.648438 5.570312 -6.34375 5.8125 -5.9375 L 5.8125 -7 L 6.953125 -7 Z M 6.953125 -0.875 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-5">
+<path style="stroke:none;" d="M 7.1875 -3.78125 L 7.1875 -3.21875 L 1.90625 -3.21875 C 1.957031 -2.425781 2.195312 -1.820312 2.625 -1.40625 C 3.050781 -1 3.644531 -0.796875 4.40625 -0.796875 C 4.84375 -0.796875 5.269531 -0.847656 5.6875 -0.953125 C 6.101562 -1.066406 6.515625 -1.226562 6.921875 -1.4375 L 6.921875 -0.359375 C 6.515625 -0.179688 6.09375 -0.046875 5.65625 0.046875 C 5.21875 0.140625 4.78125 0.1875 4.34375 0.1875 C 3.21875 0.1875 2.328125 -0.132812 1.671875 -0.78125 C 1.023438 -1.4375 0.703125 -2.320312 0.703125 -3.4375 C 0.703125 -4.582031 1.007812 -5.488281 1.625 -6.15625 C 2.25 -6.832031 3.085938 -7.171875 4.140625 -7.171875 C 5.078125 -7.171875 5.816406 -6.863281 6.359375 -6.25 C 6.910156 -5.644531 7.1875 -4.820312 7.1875 -3.78125 Z M 6.046875 -4.125 C 6.035156 -4.75 5.859375 -5.25 5.515625 -5.625 C 5.171875 -6 4.71875 -6.1875 4.15625 -6.1875 C 3.507812 -6.1875 2.992188 -6.003906 2.609375 -5.640625 C 2.222656 -5.285156 2 -4.78125 1.9375 -4.125 Z M 6.046875 -4.125 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-6">
+<path style="stroke:none;" d=""/>
+</symbol>
+<symbol overflow="visible" id="glyph0-7">
+<path style="stroke:none;" d="M 5.8125 -5.9375 L 5.8125 -9.71875 L 6.953125 -9.71875 L 6.953125 0 L 5.8125 0 L 5.8125 -1.046875 C 5.570312 -0.628906 5.265625 -0.316406 4.890625 -0.109375 C 4.523438 0.0859375 4.082031 0.1875 3.5625 0.1875 C 2.71875 0.1875 2.03125 -0.148438 1.5 -0.828125 C 0.96875 -1.503906 0.703125 -2.394531 0.703125 -3.5 C 0.703125 -4.59375 0.96875 -5.476562 1.5 -6.15625 C 2.03125 -6.832031 2.71875 -7.171875 3.5625 -7.171875 C 4.082031 -7.171875 4.523438 -7.066406 4.890625 -6.859375 C 5.265625 -6.660156 5.570312 -6.351562 5.8125 -5.9375 Z M 1.890625 -3.5 C 1.890625 -2.644531 2.0625 -1.976562 2.40625 -1.5 C 2.757812 -1.019531 3.238281 -0.78125 3.84375 -0.78125 C 4.457031 -0.78125 4.9375 -1.019531 5.28125 -1.5 C 5.632812 -1.976562 5.8125 -2.644531 5.8125 -3.5 C 5.8125 -4.34375 5.632812 -5.003906 5.28125 -5.484375 C 4.9375 -5.960938 4.457031 -6.203125 3.84375 -6.203125 C 3.238281 -6.203125 2.757812 -5.960938 2.40625 -5.484375 C 2.0625 -5.003906 1.890625 -4.34375 1.890625 -3.5 Z M 1.890625 -3.5 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-8">
+<path style="stroke:none;" d="M 2.34375 -8.984375 L 2.34375 -7 L 4.71875 -7 L 4.71875 -6.109375 L 2.34375 -6.109375 L 2.34375 -2.3125 C 2.34375 -1.738281 2.421875 -1.367188 2.578125 -1.203125 C 2.734375 -1.046875 3.050781 -0.96875 3.53125 -0.96875 L 4.71875 -0.96875 L 4.71875 0 L 3.53125 0 C 2.644531 0 2.03125 -0.164062 1.6875 -0.5 C 1.351562 -0.832031 1.1875 -1.4375 1.1875 -2.3125 L 1.1875 -6.109375 L 0.34375 -6.109375 L 0.34375 -7 L 1.1875 -7 L 1.1875 -8.984375 Z M 2.34375 -8.984375 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-9">
+<path style="stroke:none;" d="M 3.96875 -9.703125 C 3.40625 -8.742188 2.988281 -7.796875 2.71875 -6.859375 C 2.445312 -5.929688 2.3125 -4.984375 2.3125 -4.015625 C 2.3125 -3.054688 2.445312 -2.101562 2.71875 -1.15625 C 3 -0.21875 3.414062 0.726562 3.96875 1.6875 L 2.96875 1.6875 C 2.34375 0.707031 1.875 -0.253906 1.5625 -1.203125 C 1.25 -2.148438 1.09375 -3.085938 1.09375 -4.015625 C 1.09375 -4.941406 1.25 -5.875 1.5625 -6.8125 C 1.875 -7.757812 2.34375 -8.722656 2.96875 -9.703125 Z M 3.96875 -9.703125 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-10">
+<path style="stroke:none;" d="M 1.59375 -1.0625 L 3.65625 -1.0625 L 3.65625 -8.171875 L 1.40625 -7.734375 L 1.40625 -8.875 L 3.640625 -9.328125 L 4.90625 -9.328125 L 4.90625 -1.0625 L 6.953125 -1.0625 L 6.953125 0 L 1.59375 0 Z M 1.59375 -1.0625 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-11">
+<path style="stroke:none;" d="M 1.03125 -9.703125 L 2.03125 -9.703125 C 2.65625 -8.722656 3.117188 -7.757812 3.421875 -6.8125 C 3.734375 -5.875 3.890625 -4.941406 3.890625 -4.015625 C 3.890625 -3.085938 3.734375 -2.148438 3.421875 -1.203125 C 3.117188 -0.253906 2.65625 0.707031 2.03125 1.6875 L 1.03125 1.6875 C 1.582031 0.726562 1.992188 -0.21875 2.265625 -1.15625 C 2.535156 -2.101562 2.671875 -3.054688 2.671875 -4.015625 C 2.671875 -4.984375 2.535156 -5.929688 2.265625 -6.859375 C 1.992188 -7.796875 1.582031 -8.742188 1.03125 -9.703125 Z M 1.03125 -9.703125 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-12">
+<path style="stroke:none;" d="M 6.234375 -3.5 C 6.234375 -4.34375 6.054688 -5.003906 5.703125 -5.484375 C 5.359375 -5.960938 4.882812 -6.203125 4.28125 -6.203125 C 3.664062 -6.203125 3.179688 -5.960938 2.828125 -5.484375 C 2.484375 -5.003906 2.3125 -4.34375 2.3125 -3.5 C 2.3125 -2.644531 2.484375 -1.976562 2.828125 -1.5 C 3.179688 -1.019531 3.664062 -0.78125 4.28125 -0.78125 C 4.882812 -0.78125 5.359375 -1.019531 5.703125 -1.5 C 6.054688 -1.976562 6.234375 -2.644531 6.234375 -3.5 Z M 2.3125 -5.9375 C 2.5625 -6.351562 2.867188 -6.660156 3.234375 -6.859375 C 3.597656 -7.066406 4.039062 -7.171875 4.5625 -7.171875 C 5.40625 -7.171875 6.09375 -6.832031 6.625 -6.15625 C 7.15625 -5.476562 7.421875 -4.59375 7.421875 -3.5 C 7.421875 -2.394531 7.15625 -1.503906 6.625 -0.828125 C 6.09375 -0.148438 5.40625 0.1875 4.5625 0.1875 C 4.039062 0.1875 3.597656 0.0859375 3.234375 -0.109375 C 2.867188 -0.316406 2.5625 -0.628906 2.3125 -1.046875 L 2.3125 0 L 1.15625 0 L 1.15625 -9.71875 L 2.3125 -9.71875 Z M 2.3125 -5.9375 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-13">
+<path style="stroke:none;" d="M 2.453125 -1.0625 L 6.859375 -1.0625 L 6.859375 0 L 0.9375 0 L 0.9375 -1.0625 C 1.414062 -1.5625 2.066406 -2.226562 2.890625 -3.0625 C 3.722656 -3.894531 4.242188 -4.429688 4.453125 -4.671875 C 4.859375 -5.128906 5.140625 -5.515625 5.296875 -5.828125 C 5.460938 -6.140625 5.546875 -6.445312 5.546875 -6.75 C 5.546875 -7.25 5.367188 -7.65625 5.015625 -7.96875 C 4.671875 -8.28125 4.21875 -8.4375 3.65625 -8.4375 C 3.257812 -8.4375 2.84375 -8.363281 2.40625 -8.21875 C 1.96875 -8.082031 1.5 -7.878906 1 -7.609375 L 1 -8.875 C 1.507812 -9.082031 1.984375 -9.238281 2.421875 -9.34375 C 2.867188 -9.445312 3.273438 -9.5 3.640625 -9.5 C 4.609375 -9.5 5.378906 -9.253906 5.953125 -8.765625 C 6.523438 -8.285156 6.8125 -7.640625 6.8125 -6.828125 C 6.8125 -6.453125 6.738281 -6.09375 6.59375 -5.75 C 6.445312 -5.40625 6.1875 -5 5.8125 -4.53125 C 5.707031 -4.40625 5.375 -4.054688 4.8125 -3.484375 C 4.257812 -2.910156 3.472656 -2.101562 2.453125 -1.0625 Z M 2.453125 -1.0625 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-14">
+<path style="stroke:none;" d="M 5.671875 -6.796875 L 5.671875 -5.703125 C 5.347656 -5.867188 5.007812 -5.992188 4.65625 -6.078125 C 4.300781 -6.160156 3.9375 -6.203125 3.5625 -6.203125 C 3 -6.203125 2.570312 -6.113281 2.28125 -5.9375 C 2 -5.769531 1.859375 -5.507812 1.859375 -5.15625 C 1.859375 -4.882812 1.957031 -4.671875 2.15625 -4.515625 C 2.363281 -4.367188 2.773438 -4.226562 3.390625 -4.09375 L 3.78125 -4 C 4.601562 -3.832031 5.1875 -3.585938 5.53125 -3.265625 C 5.875 -2.941406 6.046875 -2.5 6.046875 -1.9375 C 6.046875 -1.28125 5.785156 -0.757812 5.265625 -0.375 C 4.753906 0 4.050781 0.1875 3.15625 0.1875 C 2.78125 0.1875 2.390625 0.148438 1.984375 0.078125 C 1.578125 0.00390625 1.144531 -0.101562 0.6875 -0.25 L 0.6875 -1.4375 C 1.113281 -1.21875 1.53125 -1.050781 1.9375 -0.9375 C 2.351562 -0.832031 2.765625 -0.78125 3.171875 -0.78125 C 3.710938 -0.78125 4.128906 -0.875 4.421875 -1.0625 C 4.710938 -1.25 4.859375 -1.507812 4.859375 -1.84375 C 4.859375 -2.15625 4.753906 -2.394531 4.546875 -2.5625 C 4.335938 -2.726562 3.875 -2.890625 3.15625 -3.046875 L 2.765625 -3.140625 C 2.046875 -3.285156 1.53125 -3.515625 1.21875 -3.828125 C 0.90625 -4.140625 0.75 -4.566406 0.75 -5.109375 C 0.75 -5.765625 0.976562 -6.269531 1.4375 -6.625 C 1.90625 -6.988281 2.570312 -7.171875 3.4375 -7.171875 C 3.851562 -7.171875 4.25 -7.140625 4.625 -7.078125 C 5 -7.015625 5.347656 -6.921875 5.671875 -6.796875 Z M 5.671875 -6.796875 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-15">
+<path style="stroke:none;" d="M 3.921875 -6.1875 C 3.304688 -6.1875 2.816406 -5.945312 2.453125 -5.46875 C 2.097656 -4.988281 1.921875 -4.332031 1.921875 -3.5 C 1.921875 -2.65625 2.097656 -1.992188 2.453125 -1.515625 C 2.804688 -1.035156 3.296875 -0.796875 3.921875 -0.796875 C 4.535156 -0.796875 5.019531 -1.035156 5.375 -1.515625 C 5.726562 -2.003906 5.90625 -2.664062 5.90625 -3.5 C 5.90625 -4.320312 5.726562 -4.972656 5.375 -5.453125 C 5.019531 -5.941406 4.535156 -6.1875 3.921875 -6.1875 Z M 3.921875 -7.171875 C 4.921875 -7.171875 5.703125 -6.84375 6.265625 -6.1875 C 6.835938 -5.539062 7.125 -4.644531 7.125 -3.5 C 7.125 -2.351562 6.835938 -1.453125 6.265625 -0.796875 C 5.703125 -0.140625 4.921875 0.1875 3.921875 0.1875 C 2.910156 0.1875 2.117188 -0.140625 1.546875 -0.796875 C 0.984375 -1.453125 0.703125 -2.351562 0.703125 -3.5 C 0.703125 -4.644531 0.984375 -5.539062 1.546875 -6.1875 C 2.117188 -6.84375 2.910156 -7.171875 3.921875 -7.171875 Z M 3.921875 -7.171875 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-16">
+<path style="stroke:none;" d="M 1.09375 -2.765625 L 1.09375 -7 L 2.234375 -7 L 2.234375 -2.8125 C 2.234375 -2.144531 2.363281 -1.644531 2.625 -1.3125 C 2.882812 -0.976562 3.269531 -0.8125 3.78125 -0.8125 C 4.40625 -0.8125 4.894531 -1.007812 5.25 -1.40625 C 5.613281 -1.800781 5.796875 -2.34375 5.796875 -3.03125 L 5.796875 -7 L 6.953125 -7 L 6.953125 0 L 5.796875 0 L 5.796875 -1.078125 C 5.515625 -0.648438 5.191406 -0.332031 4.828125 -0.125 C 4.460938 0.0820312 4.035156 0.1875 3.546875 0.1875 C 2.742188 0.1875 2.132812 -0.0625 1.71875 -0.5625 C 1.300781 -1.0625 1.09375 -1.796875 1.09375 -2.765625 Z M 3.984375 -7.171875 Z M 3.984375 -7.171875 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-17">
+<path style="stroke:none;" d="M 5.265625 -5.921875 C 5.128906 -5.992188 4.984375 -6.046875 4.828125 -6.078125 C 4.679688 -6.117188 4.519531 -6.140625 4.34375 -6.140625 C 3.6875 -6.140625 3.179688 -5.925781 2.828125 -5.5 C 2.484375 -5.082031 2.3125 -4.476562 2.3125 -3.6875 L 2.3125 0 L 1.15625 0 L 1.15625 -7 L 2.3125 -7 L 2.3125 -5.90625 C 2.5625 -6.332031 2.878906 -6.648438 3.265625 -6.859375 C 3.648438 -7.066406 4.117188 -7.171875 4.671875 -7.171875 C 4.753906 -7.171875 4.84375 -7.164062 4.9375 -7.15625 C 5.03125 -7.144531 5.132812 -7.128906 5.25 -7.109375 Z M 5.265625 -5.921875 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-18">
+<path style="stroke:none;" d="M 6.25 -6.734375 L 6.25 -5.65625 C 5.914062 -5.832031 5.585938 -5.960938 5.265625 -6.046875 C 4.941406 -6.140625 4.613281 -6.1875 4.28125 -6.1875 C 3.53125 -6.1875 2.945312 -5.953125 2.53125 -5.484375 C 2.125 -5.015625 1.921875 -4.351562 1.921875 -3.5 C 1.921875 -2.644531 2.125 -1.976562 2.53125 -1.5 C 2.945312 -1.03125 3.53125 -0.796875 4.28125 -0.796875 C 4.613281 -0.796875 4.941406 -0.835938 5.265625 -0.921875 C 5.585938 -1.015625 5.914062 -1.148438 6.25 -1.328125 L 6.25 -0.265625 C 5.925781 -0.117188 5.59375 -0.0078125 5.25 0.0625 C 4.90625 0.144531 4.539062 0.1875 4.15625 0.1875 C 3.09375 0.1875 2.25 -0.144531 1.625 -0.8125 C 1.007812 -1.476562 0.703125 -2.375 0.703125 -3.5 C 0.703125 -4.632812 1.015625 -5.53125 1.640625 -6.1875 C 2.273438 -6.84375 3.132812 -7.171875 4.21875 -7.171875 C 4.570312 -7.171875 4.914062 -7.132812 5.25 -7.0625 C 5.59375 -6.988281 5.925781 -6.878906 6.25 -6.734375 Z M 6.25 -6.734375 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-19">
+<path style="stroke:none;" d="M 2.3125 -1.046875 L 2.3125 2.65625 L 1.15625 2.65625 L 1.15625 -7 L 2.3125 -7 L 2.3125 -5.9375 C 2.5625 -6.351562 2.867188 -6.660156 3.234375 -6.859375 C 3.597656 -7.066406 4.039062 -7.171875 4.5625 -7.171875 C 5.40625 -7.171875 6.09375 -6.832031 6.625 -6.15625 C 7.15625 -5.476562 7.421875 -4.59375 7.421875 -3.5 C 7.421875 -2.394531 7.15625 -1.503906 6.625 -0.828125 C 6.09375 -0.148438 5.40625 0.1875 4.5625 0.1875 C 4.039062 0.1875 3.597656 0.0859375 3.234375 -0.109375 C 2.867188 -0.316406 2.5625 -0.628906 2.3125 -1.046875 Z M 6.234375 -3.5 C 6.234375 -4.34375 6.054688 -5.003906 5.703125 -5.484375 C 5.359375 -5.960938 4.882812 -6.203125 4.28125 -6.203125 C 3.664062 -6.203125 3.179688 -5.960938 2.828125 -5.484375 C 2.484375 -5.003906 2.3125 -4.34375 2.3125 -3.5 C 2.3125 -2.644531 2.484375 -1.976562 2.828125 -1.5 C 3.179688 -1.019531 3.664062 -0.78125 4.28125 -0.78125 C 4.882812 -0.78125 5.359375 -1.019531 5.703125 -1.5 C 6.054688 -1.976562 6.234375 -2.644531 6.234375 -3.5 Z M 6.234375 -3.5 "/>
+</symbol>
+<symbol overflow="visible" id="glyph0-20">
+<path style="stroke:none;" d="M 4.0625 -8.5 C 3.414062 -8.5 2.925781 -8.175781 2.59375 -7.53125 C 2.269531 -6.894531 2.109375 -5.9375 2.109375 -4.65625 C 2.109375 -3.375 2.269531 -2.410156 2.59375 -1.765625 C 2.925781 -1.128906 3.414062 -0.8125 4.0625 -0.8125 C 4.71875 -0.8125 5.207031 -1.128906 5.53125 -1.765625 C 5.863281 -2.410156 6.03125 -3.375 6.03125 -4.65625 C 6.03125 -5.9375 5.863281 -6.894531 5.53125 -7.53125 C 5.207031 -8.175781 4.71875 -8.5 4.0625 -8.5 Z M 4.0625 -9.5 C 5.113281 -9.5 5.914062 -9.082031 6.46875 -8.25 C 7.019531 -7.425781 7.296875 -6.226562 7.296875 -4.65625 C 7.296875 -3.082031 7.019531 -1.878906 6.46875 -1.046875 C 5.914062 -0.222656 5.113281 0.1875 4.0625 0.1875 C 3.019531 0.1875 2.222656 -0.222656 1.671875 -1.046875 C 1.117188 -1.878906 0.84375 -3.082031 0.84375 -4.65625 C 0.84375 -6.226562 1.117188 -7.425781 1.671875 -8.25 C 2.222656 -9.082031 3.019531 -9.5 4.0625 -9.5 Z M 4.0625 -9.5 "/>
+</symbol>
+</g>
+</defs>
+<g id="surface11910">
+<rect x="0" y="0" width="296" height="142" style="fill:rgb(100%,100%,100%);fill-opacity:1;stroke:none;"/>
+<path style="fill-rule:evenodd;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 16.35 11.5 L 25.75 11.5 L 25.75 18.4 L 16.35 18.4 Z M 16.35 11.5 " transform="matrix(20,0,0,20,-325,-228)"/>
+<path style="fill-rule:evenodd;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 25.919141 14.75625 C 25.919141 14.99375 25.562695 14.99375 25.562695 14.75625 C 25.562695 14.518555 25.919141 14.518555 25.919141 14.75625 " transform="matrix(20,0,0,20,-325,-228)"/>
+<path style="fill-rule:evenodd;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 18.140039 13.450586 C 18.140039 13.69707 17.770117 13.69707 17.770117 13.450586 C 17.770117 13.203906 18.140039 13.203906 18.140039 13.450586 " transform="matrix(20,0,0,20,-325,-228)"/>
+<path style="fill-rule:evenodd;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 18.116211 16.39043 C 18.116211 16.637109 17.746484 16.637109 17.746484 16.39043 C 17.746484 16.143945 18.116211 16.143945 18.116211 16.39043 " transform="matrix(20,0,0,20,-325,-228)"/>
+<path style="fill:none;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-dasharray:1,1;stroke-miterlimit:10;" d="M 18.160937 16.341406 L 25.344141 14.80293 " transform="matrix(20,0,0,20,-325,-228)"/>
+<path style="fill:none;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 25.016797 15.128711 L 25.45332 14.779687 L 24.912109 14.639844 " transform="matrix(20,0,0,20,-325,-228)"/>
+<path style="fill:none;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 18.140039 13.450586 L 25.342383 14.717383 " transform="matrix(20,0,0,20,-325,-228)"/>
+<path style="fill:none;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 24.916797 14.896484 L 25.452539 14.736914 L 25.003516 14.403906 " transform="matrix(20,0,0,20,-325,-228)"/>
+<path style="fill:none;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 25.918945 14.75625 L 30.732813 14.715039 " transform="matrix(20,0,0,20,-325,-228)"/>
+<path style="fill:none;stroke-width:0.1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;" d="M 30.34668 14.968359 L 30.844531 14.714062 L 30.342383 14.468359 " transform="matrix(20,0,0,20,-325,-228)"/>
+<g style="fill:rgb(0%,0%,0%);fill-opacity:1;">
+ <use xlink:href="#glyph0-1" x="34.101562" y="34.267687"/>
+ <use xlink:href="#glyph0-2" x="37.712674" y="34.267687"/>
+ <use xlink:href="#glyph0-3" x="50.212674" y="34.267687"/>
+ <use xlink:href="#glyph0-4" x="57.990451" y="34.267687"/>
+ <use xlink:href="#glyph0-5" x="66.046007" y="34.267687"/>
+ <use xlink:href="#glyph0-6" x="73.823785" y="34.267687"/>
+ <use xlink:href="#glyph0-7" x="77.990451" y="34.267687"/>
+ <use xlink:href="#glyph0-3" x="86.046007" y="34.267687"/>
+ <use xlink:href="#glyph0-8" x="93.823785" y="34.267687"/>
+ <use xlink:href="#glyph0-3" x="98.823785" y="34.267687"/>
+ <use xlink:href="#glyph0-6" x="106.601562" y="34.267687"/>
+ <use xlink:href="#glyph0-9" x="110.768229" y="34.267687"/>
+ <use xlink:href="#glyph0-10" x="115.768229" y="34.267687"/>
+ <use xlink:href="#glyph0-11" x="123.823785" y="34.267687"/>
+</g>
+<g style="fill:rgb(0%,0%,0%);fill-opacity:1;">
+ <use xlink:href="#glyph0-5" x="33.625" y="115.412218"/>
+ <use xlink:href="#glyph0-2" x="41.402778" y="115.412218"/>
+ <use xlink:href="#glyph0-12" x="53.902778" y="115.412218"/>
+ <use xlink:href="#glyph0-5" x="61.958333" y="115.412218"/>
+ <use xlink:href="#glyph0-7" x="69.736111" y="115.412218"/>
+ <use xlink:href="#glyph0-7" x="77.791667" y="115.412218"/>
+ <use xlink:href="#glyph0-5" x="85.847222" y="115.412218"/>
+ <use xlink:href="#glyph0-7" x="93.625" y="115.412218"/>
+ <use xlink:href="#glyph0-6" x="101.680556" y="115.412218"/>
+ <use xlink:href="#glyph0-7" x="105.847222" y="115.412218"/>
+ <use xlink:href="#glyph0-3" x="113.902778" y="115.412218"/>
+ <use xlink:href="#glyph0-8" x="121.680556" y="115.412218"/>
+ <use xlink:href="#glyph0-3" x="126.680556" y="115.412218"/>
+ <use xlink:href="#glyph0-6" x="134.458333" y="115.412218"/>
+ <use xlink:href="#glyph0-9" x="138.625" y="115.412218"/>
+ <use xlink:href="#glyph0-13" x="143.625" y="115.412218"/>
+ <use xlink:href="#glyph0-11" x="151.680556" y="115.412218"/>
+</g>
+<g style="fill:rgb(0%,0%,0%);fill-opacity:1;">
+ <use xlink:href="#glyph0-14" x="197" y="54.505968"/>
+ <use xlink:href="#glyph0-15" x="203.666667" y="54.505968"/>
+ <use xlink:href="#glyph0-16" x="211.444444" y="54.505968"/>
+ <use xlink:href="#glyph0-17" x="219.5" y="54.505968"/>
+ <use xlink:href="#glyph0-18" x="224.5" y="54.505968"/>
+ <use xlink:href="#glyph0-5" x="231.444444" y="54.505968"/>
+ <use xlink:href="#glyph0-6" x="239.222222" y="54.505968"/>
+ <use xlink:href="#glyph0-19" x="243.388889" y="54.505968"/>
+ <use xlink:href="#glyph0-3" x="251.444444" y="54.505968"/>
+ <use xlink:href="#glyph0-7" x="259.222222" y="54.505968"/>
+ <use xlink:href="#glyph0-6" x="267.277778" y="54.505968"/>
+ <use xlink:href="#glyph0-9" x="271.444444" y="54.505968"/>
+ <use xlink:href="#glyph0-20" x="276.444444" y="54.505968"/>
+ <use xlink:href="#glyph0-11" x="284.5" y="54.505968"/>
+</g>
+</g>
+</svg>
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index dcfcbd52490d..4d145bd3bd09 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -838,3 +838,5 @@ stream while it may be possible to enable and disable the embedded data stream.
The embedded data format does not need to be configured on the sensor's pads as
the format is dictated by the pixel data format in this case.
+
+.. include:: subdev-config-model.rst
diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
new file mode 100644
index 000000000000..4ddf98e3143c
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
@@ -0,0 +1,208 @@
+.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
+
+.. _media_subdev_config_model:
+
+Sub-device configuration models
+===============================
+
+A sub-device configuration model specifies in detail what the user space can
+expect from a sub-device in terms of V4L2 sub-device interface support,
+including IOCTL (including selection targets and controls) semantics.
+
+A sub-device may implement more than one configuration model at the same
+time. The implemented configuration models can be obtained from the sub-device's
+``V4L2_CID_CONFIG_MODEL`` control.
+
+.. _media_subdev_config_model_common_raw_sensor:
+
+Common raw camera sensor model
+------------------------------
+
+The common raw camera sensor model defines a set of enumeration and
+configuration interfaces (formats, selections etc.) that cover the vast majority
+of funcitionality of raw camera sensors. Not all of the interfaces are
+necessarily offered by all drivers.
+
+A sub-device complies with the common raw sensor model if the
+``V4L2_CONFIG_MODEL_COMMON_RAW`` bit is set in the ``V4L2_CID_CONFIG_MODEL``
+control of the sub-device.
+
+The common raw camera sensor model is aligned with
+:ref:`media_using_camera_sensor_drivers`. Please refer to that regarding aspects
+not specified here.
+
+Each camera sensor implementing the common raw sensor model exposes a single
+V4L2 sub-device. The sub-device contains a single source pad (0) and two or more
+internal pads: an image data internal pad (1) and optionally an embedded data
+pad (2). Additionally, further internal pads may be supported for other
+features, in which case they are documented separately for the given device.
+
+This is shown in :ref:`media_subdev_config_model_common_raw_sensor_subdev`.
+
+.. _media_subdev_config_model_common_raw_sensor_subdev:
+
+.. kernel-figure:: common-raw-sensor.svg
+ :alt: common-raw-sensor.svg
+ :align: center
+
+ **Common raw sensor sub-device**
+
+Routes
+^^^^^^
+
+A sub-device conforming to common raw camera sensor model implements the
+following routes.
+
+.. flat-table:: Routes
+ :header-rows: 1
+
+ * - Sink pad/stream
+ - Source pad/stream
+ - Static (X/M(aybe)/-)
+ - Mandatory (X/-)
+ - Synopsis
+ * - 1/0
+ - 0/0
+ - X
+ - X
+ - Image data
+ * - 2/0
+ - 0/1
+ - M
+ - \-
+ - Embedded data
+
+Some devices do not support the embedded data stream, others do support it and
+in some of the latter, it can be turned on and off before streaming is started.
+
+Sensor pixel array size, cropping and binning
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The sensor's pixel array is divided into one or more areas. The areas around the
+edge of the pixel array, usually one or more sides, may contain optical black
+pixels, dummy pixels and other non-image pixels. The entire pixel array size is
+conveyed by the format on (pad, stream) pair 1/0.
+
+A rectangle within the pixel array contains the visible pixels. Capturing the
+non-visible pixels outside the visible pixel area may be supported by the
+sensor. The visible pixel area corresponds to the ``V4L2_SEL_TGT_CROP_DEFAULT``
+selection target on (pad, stream) pair 1/0.
+
+Sensors can perform multiple operations that affect the output image size. First
+of these is the analogue crop. Analogue crop limits the area of the pixel array
+which the sensor will read, affecting sensor timing as well. The granularity of
+the analogue crop configuration varies greatly across sensors: some sensors
+support only a few different analogue crop configurations whereas others may
+support anything divisible by a given number of pixels. The analogue crop
+configuration corresponds the ``V4L2_SEL_TGT_CROP`` selection target on (pad,
+stream) pair 1/0. The default analogue crop rectangle corresponds to the visible
+pixel area.
+
+In the next step, binning is performed on the image data read from camera
+sensor's pixel array, as determined by the analogue crop configuration. Enabling
+binning will effectively result in an image smaller than the original by given
+binning factors horizontally and vertically. Typical values are 1/2 and 1/3 but
+others may well be supported by the hardware as well.
+
+Sub-sampling follows binning. Sub-sampling, like binning, reduces the size of
+the image by including only a subset of samples read from the sensor's pixel
+matrix, typically every n'th pixel horizontally and vertically, taking the
+sensor's colour pattern into account. Sub-sampling is generally configurable
+separately horizontally and vertically.
+
+Binning and sub-sampling are configured using the ``V4L2_SEL_TGT_COMPOSE``
+rectangle, relative to the analogue crop rectangle, on (pad, stream) pair
+1/0. It depends on the driver which of these operations are being used to
+achieve the resulting size.
+
+The digital crop operation takes place after binning and sub-sampling. It is
+configured by setting the ``V4L2_SEL_TGT_CROP`` rectangle on (pad, stream) pair
+0/0. The resulting image size is further output by the sensor.
+
+The sensor's output mbus code is configured by setting the format on the (pad,
+stream) pair 0/0. When setting the format, always use the same width and height
+as for the digital crop setting.
+
+Drivers may only support some of even none of these configurations, in which
+case they do not expose the corresponding selection rectangles. If any selection
+targets are omitted, the further selection rectangle or format is instead
+related to the previous implemented selection rectangle. For instance, if the
+sensor supports binning but not analogue crop, then the binning configuration
+(``V4L2_SEL_TGT_COMPOSE`` selection target) is done in relation to the visible
+pixel area (``V4L2_SEL_TGT_CROP_DEFAULT`` selection target).
+
+Also refer to :ref:`Selection targets <v4l2-selection-targets-table>`.
+
+.. flat-table:: Selection targets on pads
+ :header-rows: 1
+
+ * - Pad/Stream
+ - Selection target/format
+ - Mandatory (X/-)
+ - Modifiable (X/-)
+ - Synopsis
+ * - 1/0
+ - Format
+ - X
+ - \-
+ - Image data format. The width and the height fields indicates the full
+ size of the pixel array, including non-visible pixels. The media bus
+ code of this format reflects the native pixel depth of the sensor.
+ * - 1/0
+ - ``V4L2_SEL_TGT_CROP_DEFAULT``
+ - X
+ - \
+ - The visible pixel area. This rectangle is relative to the format on the
+ same (pad, stream).
+ * - 1/0
+ - ``V4L2_SEL_TGT_CROP``
+ - \-
+ - X
+ - Analogue crop. Analogue crop typically has a coarse granularity. This
+ rectangle is relative to the format on the same (pad, stream).
+ * - 1/0
+ - ``V4L2_SEL_TGT_COMPOSE``
+ - \-
+ - X
+ - Binning and sub-sampling. This rectangle is relative to the
+ ``V4L2_SEL_TGT_CROP`` rectangle on the same (pad, stream). The
+ combination of binning and sub-sampling is configured using this
+ selection target.
+ * - 2/0
+ - Format
+ - X
+ - \-
+ - Embedded data format.
+ * - 0/0
+ - ``V4L2_SEL_TGT_CROP``
+ - \-
+ - X
+ - Digital crop. This rectangle is relative to the ``V4L2_SEL_TGT_COMPOSE``
+ rectangle on (pad, stream) pair 1/0.
+ * - 0/0
+ - Format
+ - X
+ - X
+ - Image data source format. Always assign the width and height fields of
+ the format to the same values than for the ``V4L2_SEL_TGT_CROP``
+ rectangle on (pad, stream) pair 0/0. The media bus code reflects the
+ pixel data output of the sensor.
+ * - 0/1
+ - Format
+ - X
+ - \-
+ - Embedded data source format.
+
+Embedded data
+^^^^^^^^^^^^^
+
+The embedded data stream is produced by the sensor when the corresponding route
+is enabled. The embedded data route may also be immutable or not exist at all,
+in case the sensor (or the driver) does not support it.
+
+Generally the sensor embedded data width is determined by the width of the image
+data whereas the number of lines are constant for the embedded data. The user
+space may obtain the size of the embedded data once the image data size on the
+source pad has been configured.
+
+Also see :ref:`media_using_camera_sensor_drivers_embedded_data`.
--
2.39.5
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [RFC v3 5/9] media: Documentation: Add scaling and post-scaler crop for common raw
2024-11-29 9:51 [RFC v3 0/9] Sub-device configuration models Sakari Ailus
` (3 preceding siblings ...)
2024-11-29 9:51 ` [RFC v3 4/9] media: Documentation: Add subdev configuration models, raw sensor model Sakari Ailus
@ 2024-11-29 9:51 ` Sakari Ailus
2024-12-04 11:25 ` Jacopo Mondi
2024-11-29 9:51 ` [RFC v3 6/9] media: uapi: Add V4L2_CID_CONFIG_MODEL control Sakari Ailus
` (4 subsequent siblings)
9 siblings, 1 reply; 25+ messages in thread
From: Sakari Ailus @ 2024-11-29 9:51 UTC (permalink / raw)
To: linux-media
Cc: hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Document scaling and post-scaler digital crop operations for the common
raw sensor model.
Signedg-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
.../media/v4l/subdev-config-model.rst | 19 +++++++++++++++----
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
index 4ddf98e3143c..1ae20800f34b 100644
--- a/Documentation/userspace-api/media/v4l/subdev-config-model.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
@@ -119,9 +119,13 @@ The digital crop operation takes place after binning and sub-sampling. It is
configured by setting the ``V4L2_SEL_TGT_CROP`` rectangle on (pad, stream) pair
0/0. The resulting image size is further output by the sensor.
+The digital scaling operation is performed after the digital crop. It is
+configured by setting the ``V4L2_SEL_TGT_COMPOSE`` rectangle on (pad, stream) pair
+0/0, relative to the digital crop.
+
The sensor's output mbus code is configured by setting the format on the (pad,
-stream) pair 0/0. When setting the format, always use the same width and height
-as for the digital crop setting.
+stream) pair 0/0. The width and height fields are used to configure post-scaler
+digital crop, affecting the right side and the bottom of the frame.
Drivers may only support some of even none of these configurations, in which
case they do not expose the corresponding selection rectangles. If any selection
@@ -179,12 +183,19 @@ Also refer to :ref:`Selection targets <v4l2-selection-targets-table>`.
- X
- Digital crop. This rectangle is relative to the ``V4L2_SEL_TGT_COMPOSE``
rectangle on (pad, stream) pair 1/0.
+ * - 0/0
+ - ``V4L2_SEL_TGT_COMPOSE``
+ - \-
+ - X
+ - Scaling. This rectangle is relative to the ``V4L2_SEL_TGT_CROP``
+ rectangle on (pad, stream) pair 0/0.
* - 0/0
- Format
- X
- X
- - Image data source format. Always assign the width and height fields of
- the format to the same values than for the ``V4L2_SEL_TGT_CROP``
+ - Image data source format and post-scaler crop. The width and height
+ fields of the format, used to configure post-scaler crop on the right
+ and bottom edges of the image, are related to the ``V4L2_SEL_TGT_COMPOSE``
rectangle on (pad, stream) pair 0/0. The media bus code reflects the
pixel data output of the sensor.
* - 0/1
--
2.39.5
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [RFC v3 6/9] media: uapi: Add V4L2_CID_CONFIG_MODEL control
2024-11-29 9:51 [RFC v3 0/9] Sub-device configuration models Sakari Ailus
` (4 preceding siblings ...)
2024-11-29 9:51 ` [RFC v3 5/9] media: Documentation: Add scaling and post-scaler crop for common raw Sakari Ailus
@ 2024-11-29 9:51 ` Sakari Ailus
2024-12-05 8:43 ` Jacopo Mondi
2024-11-29 9:51 ` [RFC v3 7/9] media: uapi: Add V4L2_CID_BINNING control for binning configuration Sakari Ailus
` (3 subsequent siblings)
9 siblings, 1 reply; 25+ messages in thread
From: Sakari Ailus @ 2024-11-29 9:51 UTC (permalink / raw)
To: linux-media
Cc: hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Add the V4L2_CID_CONFIG_MODEL control for the configuration model.
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
.../userspace-api/media/v4l/ext-ctrls-image-process.rst | 4 ++++
drivers/media/v4l2-core/v4l2-ctrls-defs.c | 5 +++++
include/uapi/linux/v4l2-controls.h | 3 +++
3 files changed, 12 insertions(+)
diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-image-process.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-image-process.rst
index 27803dca8d3e..928e8e3eed7f 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-image-process.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-image-process.rst
@@ -55,3 +55,7 @@ Image Process Control IDs
control value divided by e.g. 0x100, meaning that to get no
digital gain the control value needs to be 0x100. The no-gain
configuration is also typically the default.
+
+``V4L2_CID_CONFIG_MODEL (bitmask)``
+ Which configuration models the sub-device supports. Please see
+ :ref:`media_subdev_config_model`.
diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
index 1ea52011247a..24c9c25e20d1 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
@@ -1164,6 +1164,7 @@ const char *v4l2_ctrl_get_name(u32 id)
case V4L2_CID_TEST_PATTERN: return "Test Pattern";
case V4L2_CID_DEINTERLACING_MODE: return "Deinterlacing Mode";
case V4L2_CID_DIGITAL_GAIN: return "Digital Gain";
+ case V4L2_CID_CONFIG_MODEL: return "Sub-device configuration model";
/* DV controls */
/* Keep the order of the 'case's the same as in v4l2-controls.h! */
@@ -1481,6 +1482,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
case V4L2_CID_DV_RX_POWER_PRESENT:
*type = V4L2_CTRL_TYPE_BITMASK;
break;
+ case V4L2_CID_CONFIG_MODEL:
+ *flags |= V4L2_CTRL_FLAG_READ_ONLY;
+ *type = V4L2_CTRL_TYPE_BITMASK;
+ break;
case V4L2_CID_MIN_BUFFERS_FOR_CAPTURE:
case V4L2_CID_MIN_BUFFERS_FOR_OUTPUT:
*type = V4L2_CTRL_TYPE_INTEGER;
diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
index 974fd254e573..0152240229ab 100644
--- a/include/uapi/linux/v4l2-controls.h
+++ b/include/uapi/linux/v4l2-controls.h
@@ -1225,6 +1225,9 @@ enum v4l2_jpeg_chroma_subsampling {
#define V4L2_CID_TEST_PATTERN (V4L2_CID_IMAGE_PROC_CLASS_BASE + 3)
#define V4L2_CID_DEINTERLACING_MODE (V4L2_CID_IMAGE_PROC_CLASS_BASE + 4)
#define V4L2_CID_DIGITAL_GAIN (V4L2_CID_IMAGE_PROC_CLASS_BASE + 5)
+#define V4L2_CID_CONFIG_MODEL (V4L2_CID_IMAGE_PROC_CLASS_BASE + 6)
+
+#define V4L2_CID_CONFIG_MODEL_COMMON_RAW (1ULL << 0)
/* DV-class control IDs defined by V4L2 */
#define V4L2_CID_DV_CLASS_BASE (V4L2_CTRL_CLASS_DV | 0x900)
--
2.39.5
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [RFC v3 7/9] media: uapi: Add V4L2_CID_BINNING control for binning configuration
2024-11-29 9:51 [RFC v3 0/9] Sub-device configuration models Sakari Ailus
` (5 preceding siblings ...)
2024-11-29 9:51 ` [RFC v3 6/9] media: uapi: Add V4L2_CID_CONFIG_MODEL control Sakari Ailus
@ 2024-11-29 9:51 ` Sakari Ailus
2024-12-04 11:33 ` Jacopo Mondi
2024-11-29 9:51 ` [RFC v3 8/9] media: uapi: Add controls for sub-sampling configuration Sakari Ailus
` (2 subsequent siblings)
9 siblings, 1 reply; 25+ messages in thread
From: Sakari Ailus @ 2024-11-29 9:51 UTC (permalink / raw)
To: linux-media
Cc: hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Add V4L2_CID_BINNING control for configuting binning and enumerating a
camera sensor's binning capabilities. The control combines horizontal and
vertical binning into a single control as the two are generally related.
New drivers should use this control to configure binning.
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
.../media/drivers/camera-sensor.rst | 10 +++++++
.../media/v4l/ext-ctrls-camera.rst | 29 +++++++++++++++++++
drivers/media/v4l2-core/v4l2-ctrls-defs.c | 2 ++
include/uapi/linux/v4l2-controls.h | 1 +
4 files changed, 42 insertions(+)
diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
index 5bc4c79d230c..c6167c91f46c 100644
--- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
+++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
@@ -107,6 +107,16 @@ values programmed by the register sequences. The default values of these
controls shall be 0 (disabled). Especially these controls shall not be inverted,
independently of the sensor's mounting rotation.
+Binning
+-------
+
+Binning has traditionally been configured using :ref:`the compose selection
+rectangle <v4l2-selection-targets-table>`. The :ref:`V4L2_CID_BINNING
+<v4l2-cid-camera-sensor-binning>` is also available for binning configuration and
+users should use it when it's available. Drivers supporting the control shall
+also support the compose rectangle, albeit the rectangle may be read-only when
+the control is present.
+
.. _media_using_camera_sensor_drivers_embedded_data:
Embedded data
diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst
index cdc515c60468..3fbab0e78719 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst
@@ -672,3 +672,32 @@ enum v4l2_scene_mode -
As modes differ for each sensor, menu items are not standardized by this
control and are left to the programmer.
+
+.. _v4l2-cid-camera-sensor-binning:
+
+``V4L2_CID_BINNING (integer menu)``
+
+ Horizontal and vertical binning factors. Binning combines several
+ horizontal, vertical or both pixel values into a single pixel. It is a way
+ to scale an image. Binning typically produces fairly good quality output.
+
+ Determines both horizontal and vertical binning factors for a camera
+ sensor. The values are encoded in the following way:
+
+.. flat-table::
+ :header-rows: 1
+ :stub-columns: 0
+
+ * - Bits
+ - Synopsis
+ * - 48--63
+ - Horizontal binning numerator.
+ * - 32--47
+ - Horizontal binning denominator.
+ * - 16--31
+ - Vertical binning numerator.
+ * - 0--15
+ - Vertical binning denominator.
+
+For instance, a value of ``0x0001000300020003`` indicates binning by 3
+(horizontally) * 3/2 (vertically).
diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
index 24c9c25e20d1..3a84051ec11c 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
@@ -1087,6 +1087,7 @@ const char *v4l2_ctrl_get_name(u32 id)
case V4L2_CID_CAMERA_ORIENTATION: return "Camera Orientation";
case V4L2_CID_CAMERA_SENSOR_ROTATION: return "Camera Sensor Rotation";
case V4L2_CID_HDR_SENSOR_MODE: return "HDR Sensor Mode";
+ case V4L2_CID_BINNING: return "Binning Factors";
/* FM Radio Modulator controls */
/* Keep the order of the 'case's the same as in v4l2-controls.h! */
@@ -1424,6 +1425,7 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
case V4L2_CID_HDR_SENSOR_MODE:
*type = V4L2_CTRL_TYPE_MENU;
break;
+ case V4L2_CID_BINNING:
case V4L2_CID_LINK_FREQ:
*type = V4L2_CTRL_TYPE_INTEGER_MENU;
break;
diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
index 0152240229ab..8d5815a058d7 100644
--- a/include/uapi/linux/v4l2-controls.h
+++ b/include/uapi/linux/v4l2-controls.h
@@ -1088,6 +1088,7 @@ enum v4l2_auto_focus_range {
#define V4L2_CID_CAMERA_SENSOR_ROTATION (V4L2_CID_CAMERA_CLASS_BASE+35)
#define V4L2_CID_HDR_SENSOR_MODE (V4L2_CID_CAMERA_CLASS_BASE+36)
+#define V4L2_CID_BINNING_FACTORS (V4L2_CID_CAMERA_CLASS_BASE+37)
/* FM Modulator class control IDs */
--
2.39.5
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [RFC v3 8/9] media: uapi: Add controls for sub-sampling configuration
2024-11-29 9:51 [RFC v3 0/9] Sub-device configuration models Sakari Ailus
` (6 preceding siblings ...)
2024-11-29 9:51 ` [RFC v3 7/9] media: uapi: Add V4L2_CID_BINNING control for binning configuration Sakari Ailus
@ 2024-11-29 9:51 ` Sakari Ailus
2024-11-29 9:51 ` [RFC v3 9/9] media: Documentation: Add binning and sub-sampling controls Sakari Ailus
2024-12-04 11:40 ` [RFC v3 0/9] Sub-device configuration models Sakari Ailus
9 siblings, 0 replies; 25+ messages in thread
From: Sakari Ailus @ 2024-11-29 9:51 UTC (permalink / raw)
To: linux-media
Cc: hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Sub-sampling is a way to decrease the data rates after the pixel array by
systematically discarding some samples, either vertically or horizontally
or both. Add two controls for the purpose and document them. The
sub-sampling configuration is taken into account in the compose rectangle.
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
.../userspace-api/media/drivers/camera-sensor.rst | 14 ++++++++++++--
.../userspace-api/media/v4l/ext-ctrls-camera.rst | 11 +++++++++++
drivers/media/v4l2-core/v4l2-ctrls-defs.c | 2 ++
include/uapi/linux/v4l2-controls.h | 2 ++
4 files changed, 27 insertions(+), 2 deletions(-)
diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
index c6167c91f46c..e65e82c5940a 100644
--- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
+++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
@@ -107,8 +107,8 @@ values programmed by the register sequences. The default values of these
controls shall be 0 (disabled). Especially these controls shall not be inverted,
independently of the sensor's mounting rotation.
-Binning
--------
+Binning and sub-sampling
+------------------------
Binning has traditionally been configured using :ref:`the compose selection
rectangle <v4l2-selection-targets-table>`. The :ref:`V4L2_CID_BINNING
@@ -117,6 +117,16 @@ users should use it when it's available. Drivers supporting the control shall
also support the compose rectangle, albeit the rectangle may be read-only when
the control is present.
+Sub-sampling is often supported as part of a camera sensor's binning
+functionality and performed after the binning operation. Sub-sampling typically
+produces quality-wise worse results than binning. Sub-sampling factors are
+independent horizontally and vertically and they are controlled using two
+controls, :ref:`V4L2_CID_SUBSAMPLING_HORIZONTAL and
+V4L2_CID_SUBSAMPLING_VERTICAL <v4l2-cid-camera-sensor-subsampling>`. In
+sub-sampling, the image size before sub-sampling is horizontally and vertically
+divided by the respective sub-sampling factors. Drivers supporting the control shall
+also reflect the sub-sampling configuration in the compose rectangle.
+
.. _media_using_camera_sensor_drivers_embedded_data:
Embedded data
diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst
index 3fbab0e78719..38fa97397793 100644
--- a/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst
+++ b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst
@@ -701,3 +701,14 @@ enum v4l2_scene_mode -
For instance, a value of ``0x0001000300020003`` indicates binning by 3
(horizontally) * 3/2 (vertically).
+
+.. _v4l2-cid-camera-sensor-subsampling:
+
+``V4L2_CID_SUBSAMPLING_HORIZONTAL`` and ``V4L2_CID_SUBSAMPLING_VERTICAL``
+(integer)
+
+ Horizontal and vertical subsampling factors.
+
+ Sub-sampling is used to downscale an image, horizontally and vertically, by
+ discarding a part of the image data. Typically sub-sampling produces lower
+ quality images than binning.
diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
index 3a84051ec11c..62d19b09fd6d 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
@@ -1088,6 +1088,8 @@ const char *v4l2_ctrl_get_name(u32 id)
case V4L2_CID_CAMERA_SENSOR_ROTATION: return "Camera Sensor Rotation";
case V4L2_CID_HDR_SENSOR_MODE: return "HDR Sensor Mode";
case V4L2_CID_BINNING: return "Binning Factors";
+ case V4L2_CID_SUBSAMPLING_HORIZONTAL: return "Sub-Sampling Factor, Horizontal";
+ case V4L2_CID_SUBSAMPLING_VERTICAL: return "Sub-Sampling Factor, Vertical";
/* FM Radio Modulator controls */
/* Keep the order of the 'case's the same as in v4l2-controls.h! */
diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
index 8d5815a058d7..e85747e2181b 100644
--- a/include/uapi/linux/v4l2-controls.h
+++ b/include/uapi/linux/v4l2-controls.h
@@ -1089,6 +1089,8 @@ enum v4l2_auto_focus_range {
#define V4L2_CID_HDR_SENSOR_MODE (V4L2_CID_CAMERA_CLASS_BASE+36)
#define V4L2_CID_BINNING_FACTORS (V4L2_CID_CAMERA_CLASS_BASE+37)
+#define V4L2_CID_SUBSAMPLING_HORIZONTAL (V4L2_CID_CAMERA_CLASS_BASE+38)
+#define V4L2_CID_SUBSAMPLING_VERTICAL (V4L2_CID_CAMERA_CLASS_BASE+39)
/* FM Modulator class control IDs */
--
2.39.5
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [RFC v3 9/9] media: Documentation: Add binning and sub-sampling controls
2024-11-29 9:51 [RFC v3 0/9] Sub-device configuration models Sakari Ailus
` (7 preceding siblings ...)
2024-11-29 9:51 ` [RFC v3 8/9] media: uapi: Add controls for sub-sampling configuration Sakari Ailus
@ 2024-11-29 9:51 ` Sakari Ailus
2024-12-04 11:40 ` [RFC v3 0/9] Sub-device configuration models Sakari Ailus
9 siblings, 0 replies; 25+ messages in thread
From: Sakari Ailus @ 2024-11-29 9:51 UTC (permalink / raw)
To: linux-media
Cc: hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Document the binning and scaling controls (V4L2_CID_BINNING and
V4L2_CID_SUBSAMPLING_{HORIZONTAL,VERTICAL}) in the common raw sensor
model.
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
.../media/v4l/subdev-config-model.rst | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)
diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
index 1ae20800f34b..06a94ff4d2b4 100644
--- a/Documentation/userspace-api/media/v4l/subdev-config-model.rst
+++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
@@ -113,7 +113,10 @@ separately horizontally and vertically.
Binning and sub-sampling are configured using the ``V4L2_SEL_TGT_COMPOSE``
rectangle, relative to the analogue crop rectangle, on (pad, stream) pair
1/0. It depends on the driver which of these operations are being used to
-achieve the resulting size.
+achieve the resulting size. Binning and sub-sampling are also directly
+configured using :ref:`V4L2_CID_BINNING <v4l2-cid-camera-sensor-binning>` and
+:ref:`V4L2_CID_SUBSAMPLING_HORIZONTAL and V4L2_CID_SUBSAMPLING_VERTICAL
+<v4l2-cid-camera-sensor-subsampling>` controls on drivers that support them.
The digital crop operation takes place after binning and sub-sampling. It is
configured by setting the ``V4L2_SEL_TGT_CROP`` rectangle on (pad, stream) pair
@@ -169,9 +172,15 @@ Also refer to :ref:`Selection targets <v4l2-selection-targets-table>`.
- \-
- X
- Binning and sub-sampling. This rectangle is relative to the
- ``V4L2_SEL_TGT_CROP`` rectangle on the same (pad, stream). The
- combination of binning and sub-sampling is configured using this
- selection target.
+ ``V4L2_SEL_TGT_CROP`` rectangle on the same (pad, stream). Binning is
+ configured using the :ref:`V4L2_CID_BINNING
+ <v4l2-cid-camera-sensor-binning>` control and sub-sampling is configured
+ using the :ref:`V4L2_CID_SUBSAMPLING_HORIZONTAL and
+ V4L2_CID_SUBSAMPLING_VERTICAL <v4l2-cid-camera-sensor-subsampling>`
+ controls on drivers that support these controls. To configure binning
+ and sub-sampling on drivers that do not support these controls, the
+ selection rectangle may be changed directly to configure the combined
+ effect on the image size.
* - 2/0
- Format
- X
--
2.39.5
^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: [RFC v3 5/9] media: Documentation: Add scaling and post-scaler crop for common raw
2024-11-29 9:51 ` [RFC v3 5/9] media: Documentation: Add scaling and post-scaler crop for common raw Sakari Ailus
@ 2024-12-04 11:25 ` Jacopo Mondi
2024-12-04 14:15 ` Sakari Ailus
0 siblings, 1 reply; 25+ messages in thread
From: Jacopo Mondi @ 2024-12-04 11:25 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Sakari
thanks for the update
On Fri, Nov 29, 2024 at 11:51:38AM +0200, Sakari Ailus wrote:
> Document scaling and post-scaler digital crop operations for the common
> raw sensor model.
> Signedg-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
> .../media/v4l/subdev-config-model.rst | 19 +++++++++++++++----
> 1 file changed, 15 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> index 4ddf98e3143c..1ae20800f34b 100644
> --- a/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> +++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> @@ -119,9 +119,13 @@ The digital crop operation takes place after binning and sub-sampling. It is
> configured by setting the ``V4L2_SEL_TGT_CROP`` rectangle on (pad, stream) pair
> 0/0. The resulting image size is further output by the sensor.
I think this last line "The resulting image size is further output by
the sensor." doesn't apply anymore now that we have [digital crop +
digital scaling + post-scaling crop].
>
> +The digital scaling operation is performed after the digital crop. It is
> +configured by setting the ``V4L2_SEL_TGT_COMPOSE`` rectangle on (pad, stream) pair
> +0/0, relative to the digital crop.
> +
"to the digital crop rectangle." maybe ?
> The sensor's output mbus code is configured by setting the format on the (pad,
> -stream) pair 0/0. When setting the format, always use the same width and height
> -as for the digital crop setting.
> +stream) pair 0/0. The width and height fields are used to configure post-scaler
> +digital crop, affecting the right side and the bottom of the frame.
>
I would introduce the (optional) presence of an additional post-scaler
digital crop step explicitly, also I get this is kind of rare feature,
isn't it ?
-------------------------------------------------------------------------------
The digital scaling operation is performed after the digital crop. It
is configured by setting the ``V4L2_SEL_TGT_COMPOSE`` rectangle on
(pad, stream) pair 0/0, relative to the digital crop rectangle.
The sensor's output mbus code is configured by setting the format on the (pad,
stream) pair 0/0. An optional post-scaler crop step might be performed by
specifying a width and height smaller than the digital scaling
rectangle. If post-scaler cropping is not supported the format's sizes
will always match the compose rectangle sizes applied on the same 0/0
(pad, stream) pair.
-------------------------------------------------------------------------------
I didn't get when we spoke about it where the "affecting the right
side and bottom of the frame" comes from when talking about
post-scaler cropping.
> Drivers may only support some of even none of these configurations, in which
> case they do not expose the corresponding selection rectangles. If any selection
> @@ -179,12 +183,19 @@ Also refer to :ref:`Selection targets <v4l2-selection-targets-table>`.
> - X
> - Digital crop. This rectangle is relative to the ``V4L2_SEL_TGT_COMPOSE``
> rectangle on (pad, stream) pair 1/0.
> + * - 0/0
> + - ``V4L2_SEL_TGT_COMPOSE``
> + - \-
> + - X
> + - Scaling. This rectangle is relative to the ``V4L2_SEL_TGT_CROP``
"Digital scaling" ? or is not necessary ?
> + rectangle on (pad, stream) pair 0/0.
> * - 0/0
> - Format
> - X
> - X
> - - Image data source format. Always assign the width and height fields of
> - the format to the same values than for the ``V4L2_SEL_TGT_CROP``
> + - Image data source format and post-scaler crop. The width and height
> + fields of the format, used to configure post-scaler crop on the right
> + and bottom edges of the image, are related to the ``V4L2_SEL_TGT_COMPOSE``
> rectangle on (pad, stream) pair 0/0. The media bus code reflects the
> pixel data output of the sensor.
> * - 0/1
> --
> 2.39.5
>
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 7/9] media: uapi: Add V4L2_CID_BINNING control for binning configuration
2024-11-29 9:51 ` [RFC v3 7/9] media: uapi: Add V4L2_CID_BINNING control for binning configuration Sakari Ailus
@ 2024-12-04 11:33 ` Jacopo Mondi
2024-12-04 14:24 ` Sakari Ailus
0 siblings, 1 reply; 25+ messages in thread
From: Jacopo Mondi @ 2024-12-04 11:33 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Sakari
On Fri, Nov 29, 2024 at 11:51:40AM +0200, Sakari Ailus wrote:
> Add V4L2_CID_BINNING control for configuting binning and enumerating a
> camera sensor's binning capabilities. The control combines horizontal and
> vertical binning into a single control as the two are generally related.
>
> New drivers should use this control to configure binning.
I thought the idea was to post-pone this (and the following 2 patches)
to when controls will be able to be tested without changing the device
configuration, to properly support TRY_FORMAT, and before that time
work with selection rectangles and let the drivers select the
"correct" binning/scaling factors deduced from the
TGT_CROP-to-TGT_COMPOSE ratio on (pad, stream) 1/0
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
> .../media/drivers/camera-sensor.rst | 10 +++++++
> .../media/v4l/ext-ctrls-camera.rst | 29 +++++++++++++++++++
> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 2 ++
> include/uapi/linux/v4l2-controls.h | 1 +
> 4 files changed, 42 insertions(+)
>
> diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> index 5bc4c79d230c..c6167c91f46c 100644
> --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> @@ -107,6 +107,16 @@ values programmed by the register sequences. The default values of these
> controls shall be 0 (disabled). Especially these controls shall not be inverted,
> independently of the sensor's mounting rotation.
>
> +Binning
> +-------
> +
> +Binning has traditionally been configured using :ref:`the compose selection
> +rectangle <v4l2-selection-targets-table>`. The :ref:`V4L2_CID_BINNING
> +<v4l2-cid-camera-sensor-binning>` is also available for binning configuration and
> +users should use it when it's available. Drivers supporting the control shall
> +also support the compose rectangle, albeit the rectangle may be read-only when
> +the control is present.
> +
> .. _media_using_camera_sensor_drivers_embedded_data:
>
> Embedded data
> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst
> index cdc515c60468..3fbab0e78719 100644
> --- a/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst
> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-camera.rst
> @@ -672,3 +672,32 @@ enum v4l2_scene_mode -
>
> As modes differ for each sensor, menu items are not standardized by this
> control and are left to the programmer.
> +
> +.. _v4l2-cid-camera-sensor-binning:
> +
> +``V4L2_CID_BINNING (integer menu)``
> +
> + Horizontal and vertical binning factors. Binning combines several
> + horizontal, vertical or both pixel values into a single pixel. It is a way
> + to scale an image. Binning typically produces fairly good quality output.
> +
> + Determines both horizontal and vertical binning factors for a camera
> + sensor. The values are encoded in the following way:
> +
> +.. flat-table::
> + :header-rows: 1
> + :stub-columns: 0
> +
> + * - Bits
> + - Synopsis
> + * - 48--63
> + - Horizontal binning numerator.
> + * - 32--47
> + - Horizontal binning denominator.
> + * - 16--31
> + - Vertical binning numerator.
> + * - 0--15
> + - Vertical binning denominator.
> +
> +For instance, a value of ``0x0001000300020003`` indicates binning by 3
> +(horizontally) * 3/2 (vertically).
> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> index 24c9c25e20d1..3a84051ec11c 100644
> --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> @@ -1087,6 +1087,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> case V4L2_CID_CAMERA_ORIENTATION: return "Camera Orientation";
> case V4L2_CID_CAMERA_SENSOR_ROTATION: return "Camera Sensor Rotation";
> case V4L2_CID_HDR_SENSOR_MODE: return "HDR Sensor Mode";
> + case V4L2_CID_BINNING: return "Binning Factors";
>
> /* FM Radio Modulator controls */
> /* Keep the order of the 'case's the same as in v4l2-controls.h! */
> @@ -1424,6 +1425,7 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
> case V4L2_CID_HDR_SENSOR_MODE:
> *type = V4L2_CTRL_TYPE_MENU;
> break;
> + case V4L2_CID_BINNING:
> case V4L2_CID_LINK_FREQ:
> *type = V4L2_CTRL_TYPE_INTEGER_MENU;
> break;
> diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
> index 0152240229ab..8d5815a058d7 100644
> --- a/include/uapi/linux/v4l2-controls.h
> +++ b/include/uapi/linux/v4l2-controls.h
> @@ -1088,6 +1088,7 @@ enum v4l2_auto_focus_range {
> #define V4L2_CID_CAMERA_SENSOR_ROTATION (V4L2_CID_CAMERA_CLASS_BASE+35)
>
> #define V4L2_CID_HDR_SENSOR_MODE (V4L2_CID_CAMERA_CLASS_BASE+36)
> +#define V4L2_CID_BINNING_FACTORS (V4L2_CID_CAMERA_CLASS_BASE+37)
>
> /* FM Modulator class control IDs */
>
> --
> 2.39.5
>
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 0/9] Sub-device configuration models
2024-11-29 9:51 [RFC v3 0/9] Sub-device configuration models Sakari Ailus
` (8 preceding siblings ...)
2024-11-29 9:51 ` [RFC v3 9/9] media: Documentation: Add binning and sub-sampling controls Sakari Ailus
@ 2024-12-04 11:40 ` Sakari Ailus
9 siblings, 0 replies; 25+ messages in thread
From: Sakari Ailus @ 2024-12-04 11:40 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
On Fri, Nov 29, 2024 at 11:51:33AM +0200, Sakari Ailus wrote:
> The compiled documentation can be found here
> <URL:https://www.retiisi.eu/~sailus/v4l2/tmp/common-raw/output/userspace-api/media/v4l/dev-subdev.html#media-subdev-config-model-common-raw-sensor>.
The patches can be also found at
<URL:https://git.retiisi.eu/?p=~sailus/linux.git;a=shortlog;h=refs/heads/metadata>.
--
Sakari Ailus
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 4/9] media: Documentation: Add subdev configuration models, raw sensor model
2024-11-29 9:51 ` [RFC v3 4/9] media: Documentation: Add subdev configuration models, raw sensor model Sakari Ailus
@ 2024-12-04 12:10 ` Jacopo Mondi
2024-12-04 13:22 ` Sakari Ailus
0 siblings, 1 reply; 25+ messages in thread
From: Jacopo Mondi @ 2024-12-04 12:10 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Sakari
On Fri, Nov 29, 2024 at 11:51:37AM +0200, Sakari Ailus wrote:
> Sub-device configuration models define what V4L2 API elements are
> available on a compliant sub-device and how do they behave.
>
> The patch also adds a model for common raw sensors.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> ---
> .../media/drivers/camera-sensor.rst | 4 +
> .../media/v4l/common-raw-sensor.dia | 441 ++++++++++++++++++
> .../media/v4l/common-raw-sensor.svg | 134 ++++++
> .../userspace-api/media/v4l/dev-subdev.rst | 2 +
> .../media/v4l/subdev-config-model.rst | 208 +++++++++
> 5 files changed, 789 insertions(+)
> create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.dia
> create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.svg
> create mode 100644 Documentation/userspace-api/media/v4l/subdev-config-model.rst
>
> diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> index bc55c861fb69..5bc4c79d230c 100644
> --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> @@ -18,6 +18,8 @@ binning functionality. The sensor drivers belong to two distinct classes, freely
> configurable and register list-based drivers, depending on how the driver
> configures this functionality.
>
> +Also see :ref:`media_subdev_config_model_common_raw_sensor`.
> +
> Freely configurable camera sensor drivers
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> @@ -105,6 +107,8 @@ values programmed by the register sequences. The default values of these
> controls shall be 0 (disabled). Especially these controls shall not be inverted,
> independently of the sensor's mounting rotation.
>
> +.. _media_using_camera_sensor_drivers_embedded_data:
> +
> Embedded data
> -------------
>
> diff --git a/Documentation/userspace-api/media/v4l/common-raw-sensor.dia b/Documentation/userspace-api/media/v4l/common-raw-sensor.dia
snip
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index dcfcbd52490d..4d145bd3bd09 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -838,3 +838,5 @@ stream while it may be possible to enable and disable the embedded data stream.
>
> The embedded data format does not need to be configured on the sensor's pads as
> the format is dictated by the pixel data format in this case.
> +
> +.. include:: subdev-config-model.rst
> diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> new file mode 100644
> index 000000000000..4ddf98e3143c
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> @@ -0,0 +1,208 @@
> +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> +
> +.. _media_subdev_config_model:
> +
> +Sub-device configuration models
> +===============================
> +
> +A sub-device configuration model specifies in detail what the user space can
> +expect from a sub-device in terms of V4L2 sub-device interface support,
> +including IOCTL (including selection targets and controls) semantics.
including IOCTLs, selection targets and controls semantics.
> +
> +A sub-device may implement more than one configuration model at the same
> +time. The implemented configuration models can be obtained from the sub-device's
> +``V4L2_CID_CONFIG_MODEL`` control.
> +
> +.. _media_subdev_config_model_common_raw_sensor:
> +
> +Common raw camera sensor model
> +------------------------------
> +
> +The common raw camera sensor model defines a set of enumeration and
> +configuration interfaces (formats, selections etc.) that cover the vast majority
> +of funcitionality of raw camera sensors. Not all of the interfaces are
s/funcitionality/functionalities
> +necessarily offered by all drivers.
> +
> +A sub-device complies with the common raw sensor model if the
> +``V4L2_CONFIG_MODEL_COMMON_RAW`` bit is set in the ``V4L2_CID_CONFIG_MODEL``
> +control of the sub-device.
> +
> +The common raw camera sensor model is aligned with
> +:ref:`media_using_camera_sensor_drivers`. Please refer to that regarding aspects
> +not specified here.
> +
> +Each camera sensor implementing the common raw sensor model exposes a single
> +V4L2 sub-device. The sub-device contains a single source pad (0) and two or more
> +internal pads: an image data internal pad (1) and optionally an embedded data
> +pad (2). Additionally, further internal pads may be supported for other
> +features, in which case they are documented separately for the given device.
> +
> +This is shown in :ref:`media_subdev_config_model_common_raw_sensor_subdev`.
> +
> +.. _media_subdev_config_model_common_raw_sensor_subdev:
> +
> +.. kernel-figure:: common-raw-sensor.svg
> + :alt: common-raw-sensor.svg
> + :align: center
> +
> + **Common raw sensor sub-device**
> +
> +Routes
> +^^^^^^
> +
> +A sub-device conforming to common raw camera sensor model implements the
> +following routes.
> +
> +.. flat-table:: Routes
> + :header-rows: 1
> +
> + * - Sink pad/stream
> + - Source pad/stream
> + - Static (X/M(aybe)/-)
> + - Mandatory (X/-)
> + - Synopsis
> + * - 1/0
> + - 0/0
> + - X
> + - X
> + - Image data
> + * - 2/0
> + - 0/1
> + - M
> + - \-
> + - Embedded data
> +
> +Some devices do not support the embedded data stream, others do support it and
> +in some of the latter, it can be turned on and off before streaming is started.
What about:
Support for the embedded data stream is optional. Device that support
the embedded data stream might allow to enable/disable the route
before the streaming is started.
> +
> +Sensor pixel array size, cropping and binning
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The sensor's pixel array is divided into one or more areas. The areas around the
> +edge of the pixel array, usually one or more sides, may contain optical black
optically ?
> +pixels, dummy pixels and other non-image pixels. The entire pixel array size is
> +conveyed by the format on (pad, stream) pair 1/0.
> +
> +A rectangle within the pixel array contains the visible pixels. Capturing the
> +non-visible pixels outside the visible pixel area may be supported by the
> +sensor. The visible pixel area corresponds to the ``V4L2_SEL_TGT_CROP_DEFAULT``
> +selection target on (pad, stream) pair 1/0.
> +
> +Sensors can perform multiple operations that affect the output image size. First
> +of these is the analogue crop. Analogue crop limits the area of the pixel array
> +which the sensor will read, affecting sensor timing as well. The granularity of
> +the analogue crop configuration varies greatly across sensors: some sensors
> +support only a few different analogue crop configurations whereas others may
> +support anything divisible by a given number of pixels. The analogue crop
> +configuration corresponds the ``V4L2_SEL_TGT_CROP`` selection target on (pad,
> +stream) pair 1/0. The default analogue crop rectangle corresponds to the visible
> +pixel area.
> +
> +In the next step, binning is performed on the image data read from camera
> +sensor's pixel array, as determined by the analogue crop configuration. Enabling
> +binning will effectively result in an image smaller than the original by given
> +binning factors horizontally and vertically. Typical values are 1/2 and 1/3 but
I thought 1/4 was more common than 1/3... Nevermind..
> +others may well be supported by the hardware as well.
> +
> +Sub-sampling follows binning. Sub-sampling, like binning, reduces the size of
> +the image by including only a subset of samples read from the sensor's pixel
> +matrix, typically every n'th pixel horizontally and vertically, taking the
> +sensor's colour pattern into account. Sub-sampling is generally configurable
> +separately horizontally and vertically.
> +
> +Binning and sub-sampling are configured using the ``V4L2_SEL_TGT_COMPOSE``
> +rectangle, relative to the analogue crop rectangle, on (pad, stream) pair
> +1/0. It depends on the driver which of these operations are being used to
> +achieve the resulting size.
I would say "It depends on the driver implementation to decide how
binning and skipping are combined to obtain the desired scaling
ratio."
> +
> +The digital crop operation takes place after binning and sub-sampling. It is
> +configured by setting the ``V4L2_SEL_TGT_CROP`` rectangle on (pad, stream) pair
> +0/0. The resulting image size is further output by the sensor.
This is not true anymore if we consider digital scaling and
post-scaler crop.
> +
> +The sensor's output mbus code is configured by setting the format on the (pad,
> +stream) pair 0/0. When setting the format, always use the same width and height
> +as for the digital crop setting.
> +
Same comment here as well.
> +Drivers may only support some of even none of these configurations, in which
s/configurations/configuration options/
> +case they do not expose the corresponding selection rectangles. If any selection
How are selection rectangles "exposed" ?
I would say "support" instead of expose.
However I'm a bit concerned what "support" (or "expose") means. We
should probably define it. I do think that driver should allow reading
all the above selection rectangles and return the correct values (if a
configuration step is not supported, sizes are simply passed forward
from the previous selection target to the next target/format). I'm
less sure about setting them, if they should error out or simply
adjust it as get_selection() would have done.
> +targets are omitted, the further selection rectangle or format is instead
> +related to the previous implemented selection rectangle. For instance, if the
> +sensor supports binning but not analogue crop, then the binning configuration
> +(``V4L2_SEL_TGT_COMPOSE`` selection target) is done in relation to the visible
> +pixel area (``V4L2_SEL_TGT_CROP_DEFAULT`` selection target).
Let alone the fact I would have used, say, digital crop as an example
of an optional feature, I think allowing to read all the possible
targets would save userspace keeping track of what target the
rectangle they want to configure refers to.
> +
> +Also refer to :ref:`Selection targets <v4l2-selection-targets-table>`.
> +
> +.. flat-table:: Selection targets on pads
> + :header-rows: 1
> +
> + * - Pad/Stream
> + - Selection target/format
> + - Mandatory (X/-)
> + - Modifiable (X/-)
> + - Synopsis
> + * - 1/0
> + - Format
> + - X
> + - \-
> + - Image data format. The width and the height fields indicates the full
> + size of the pixel array, including non-visible pixels. The media bus
> + code of this format reflects the native pixel depth of the sensor.
> + * - 1/0
> + - ``V4L2_SEL_TGT_CROP_DEFAULT``
> + - X
> + - \
> + - The visible pixel area. This rectangle is relative to the format on the
> + same (pad, stream).
> + * - 1/0
> + - ``V4L2_SEL_TGT_CROP``
> + - \-
> + - X
> + - Analogue crop. Analogue crop typically has a coarse granularity. This
> + rectangle is relative to the format on the same (pad, stream).
> + * - 1/0
> + - ``V4L2_SEL_TGT_COMPOSE``
> + - \-
> + - X
> + - Binning and sub-sampling. This rectangle is relative to the
> + ``V4L2_SEL_TGT_CROP`` rectangle on the same (pad, stream). The
> + combination of binning and sub-sampling is configured using this
> + selection target.
> + * - 2/0
> + - Format
> + - X
> + - \-
> + - Embedded data format.
> + * - 0/0
> + - ``V4L2_SEL_TGT_CROP``
> + - \-
> + - X
> + - Digital crop. This rectangle is relative to the ``V4L2_SEL_TGT_COMPOSE``
> + rectangle on (pad, stream) pair 1/0.
> + * - 0/0
> + - Format
> + - X
> + - X
> + - Image data source format. Always assign the width and height fields of
> + the format to the same values than for the ``V4L2_SEL_TGT_CROP``
> + rectangle on (pad, stream) pair 0/0. The media bus code reflects the
> + pixel data output of the sensor.
This makes me think it is intentional not to document digital
scaling/post-scaler crop in this version ?
> + * - 0/1
> + - Format
> + - X
> + - \-
> + - Embedded data source format.
> +
> +Embedded data
> +^^^^^^^^^^^^^
> +
> +The embedded data stream is produced by the sensor when the corresponding route
> +is enabled. The embedded data route may also be immutable or not exist at all,
> +in case the sensor (or the driver) does not support it.
> +
> +Generally the sensor embedded data width is determined by the width of the image
> +data whereas the number of lines are constant for the embedded data. The user
Maybe drop "The" from "The user space" ?
> +space may obtain the size of the embedded data once the image data size on the
> +source pad has been configured.
> +
> +Also see :ref:`media_using_camera_sensor_drivers_embedded_data`.
Thanks for the hard work here, I like how this new document
clearly outlines the expected interfaces exposed by drivers.
Thanks
j
> --
> 2.39.5
>
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 4/9] media: Documentation: Add subdev configuration models, raw sensor model
2024-12-04 12:10 ` Jacopo Mondi
@ 2024-12-04 13:22 ` Sakari Ailus
2024-12-04 13:37 ` Jacopo Mondi
0 siblings, 1 reply; 25+ messages in thread
From: Sakari Ailus @ 2024-12-04 13:22 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Jacopo,
Thank you for the review.
On Wed, Dec 04, 2024 at 01:10:55PM +0100, Jacopo Mondi wrote:
> Hi Sakari
>
> On Fri, Nov 29, 2024 at 11:51:37AM +0200, Sakari Ailus wrote:
> > Sub-device configuration models define what V4L2 API elements are
> > available on a compliant sub-device and how do they behave.
> >
> > The patch also adds a model for common raw sensors.
> >
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> > .../media/drivers/camera-sensor.rst | 4 +
> > .../media/v4l/common-raw-sensor.dia | 441 ++++++++++++++++++
> > .../media/v4l/common-raw-sensor.svg | 134 ++++++
> > .../userspace-api/media/v4l/dev-subdev.rst | 2 +
> > .../media/v4l/subdev-config-model.rst | 208 +++++++++
> > 5 files changed, 789 insertions(+)
> > create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.dia
> > create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.svg
> > create mode 100644 Documentation/userspace-api/media/v4l/subdev-config-model.rst
> >
> > diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > index bc55c861fb69..5bc4c79d230c 100644
> > --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > @@ -18,6 +18,8 @@ binning functionality. The sensor drivers belong to two distinct classes, freely
> > configurable and register list-based drivers, depending on how the driver
> > configures this functionality.
> >
> > +Also see :ref:`media_subdev_config_model_common_raw_sensor`.
> > +
> > Freely configurable camera sensor drivers
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > @@ -105,6 +107,8 @@ values programmed by the register sequences. The default values of these
> > controls shall be 0 (disabled). Especially these controls shall not be inverted,
> > independently of the sensor's mounting rotation.
> >
> > +.. _media_using_camera_sensor_drivers_embedded_data:
> > +
> > Embedded data
> > -------------
> >
> > diff --git a/Documentation/userspace-api/media/v4l/common-raw-sensor.dia b/Documentation/userspace-api/media/v4l/common-raw-sensor.dia
>
> snip
>
> > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > index dcfcbd52490d..4d145bd3bd09 100644
> > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > @@ -838,3 +838,5 @@ stream while it may be possible to enable and disable the embedded data stream.
> >
> > The embedded data format does not need to be configured on the sensor's pads as
> > the format is dictated by the pixel data format in this case.
> > +
> > +.. include:: subdev-config-model.rst
> > diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> > new file mode 100644
> > index 000000000000..4ddf98e3143c
> > --- /dev/null
> > +++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> > @@ -0,0 +1,208 @@
> > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > +
> > +.. _media_subdev_config_model:
> > +
> > +Sub-device configuration models
> > +===============================
> > +
> > +A sub-device configuration model specifies in detail what the user space can
> > +expect from a sub-device in terms of V4L2 sub-device interface support,
> > +including IOCTL (including selection targets and controls) semantics.
>
> including IOCTLs, selection targets and controls semantics.
I think we could as well as drop these, i.e. just mention semantics.
>
> > +
> > +A sub-device may implement more than one configuration model at the same
> > +time. The implemented configuration models can be obtained from the sub-device's
> > +``V4L2_CID_CONFIG_MODEL`` control.
> > +
> > +.. _media_subdev_config_model_common_raw_sensor:
> > +
> > +Common raw camera sensor model
> > +------------------------------
> > +
> > +The common raw camera sensor model defines a set of enumeration and
> > +configuration interfaces (formats, selections etc.) that cover the vast majority
> > +of funcitionality of raw camera sensors. Not all of the interfaces are
>
> s/funcitionality/functionalities
I'd say singular is right in this case. Maybe Kieran or Dave have an
opinion as well? :-)
>
> > +necessarily offered by all drivers.
> > +
> > +A sub-device complies with the common raw sensor model if the
> > +``V4L2_CONFIG_MODEL_COMMON_RAW`` bit is set in the ``V4L2_CID_CONFIG_MODEL``
> > +control of the sub-device.
> > +
> > +The common raw camera sensor model is aligned with
> > +:ref:`media_using_camera_sensor_drivers`. Please refer to that regarding aspects
> > +not specified here.
> > +
> > +Each camera sensor implementing the common raw sensor model exposes a single
> > +V4L2 sub-device. The sub-device contains a single source pad (0) and two or more
> > +internal pads: an image data internal pad (1) and optionally an embedded data
> > +pad (2). Additionally, further internal pads may be supported for other
> > +features, in which case they are documented separately for the given device.
> > +
> > +This is shown in :ref:`media_subdev_config_model_common_raw_sensor_subdev`.
> > +
> > +.. _media_subdev_config_model_common_raw_sensor_subdev:
> > +
> > +.. kernel-figure:: common-raw-sensor.svg
> > + :alt: common-raw-sensor.svg
> > + :align: center
> > +
> > + **Common raw sensor sub-device**
> > +
> > +Routes
> > +^^^^^^
> > +
> > +A sub-device conforming to common raw camera sensor model implements the
> > +following routes.
> > +
> > +.. flat-table:: Routes
> > + :header-rows: 1
> > +
> > + * - Sink pad/stream
> > + - Source pad/stream
> > + - Static (X/M(aybe)/-)
> > + - Mandatory (X/-)
> > + - Synopsis
> > + * - 1/0
> > + - 0/0
> > + - X
> > + - X
> > + - Image data
> > + * - 2/0
> > + - 0/1
> > + - M
> > + - \-
> > + - Embedded data
> > +
> > +Some devices do not support the embedded data stream, others do support it and
> > +in some of the latter, it can be turned on and off before streaming is started.
>
> What about:
>
> Support for the embedded data stream is optional. Device that support
> the embedded data stream might allow to enable/disable the route
> before the streaming is started.
How about:
Support for the embedded data stream is optional. Drivers supporting
the embedded data stream may allow disabling and enabling the route
when the streaming is disabled.
>
> > +
> > +Sensor pixel array size, cropping and binning
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +The sensor's pixel array is divided into one or more areas. The areas around the
> > +edge of the pixel array, usually one or more sides, may contain optical black
>
> optically ?
E.g. CCS spec refers to these as "optical black" pixels.
>
> > +pixels, dummy pixels and other non-image pixels. The entire pixel array size is
> > +conveyed by the format on (pad, stream) pair 1/0.
> > +
> > +A rectangle within the pixel array contains the visible pixels. Capturing the
> > +non-visible pixels outside the visible pixel area may be supported by the
> > +sensor. The visible pixel area corresponds to the ``V4L2_SEL_TGT_CROP_DEFAULT``
> > +selection target on (pad, stream) pair 1/0.
> > +
> > +Sensors can perform multiple operations that affect the output image size. First
> > +of these is the analogue crop. Analogue crop limits the area of the pixel array
> > +which the sensor will read, affecting sensor timing as well. The granularity of
> > +the analogue crop configuration varies greatly across sensors: some sensors
> > +support only a few different analogue crop configurations whereas others may
> > +support anything divisible by a given number of pixels. The analogue crop
> > +configuration corresponds the ``V4L2_SEL_TGT_CROP`` selection target on (pad,
> > +stream) pair 1/0. The default analogue crop rectangle corresponds to the visible
> > +pixel area.
> > +
> > +In the next step, binning is performed on the image data read from camera
> > +sensor's pixel array, as determined by the analogue crop configuration. Enabling
> > +binning will effectively result in an image smaller than the original by given
> > +binning factors horizontally and vertically. Typical values are 1/2 and 1/3 but
>
> I thought 1/4 was more common than 1/3... Nevermind..
>
> > +others may well be supported by the hardware as well.
> > +
> > +Sub-sampling follows binning. Sub-sampling, like binning, reduces the size of
> > +the image by including only a subset of samples read from the sensor's pixel
> > +matrix, typically every n'th pixel horizontally and vertically, taking the
> > +sensor's colour pattern into account. Sub-sampling is generally configurable
> > +separately horizontally and vertically.
> > +
> > +Binning and sub-sampling are configured using the ``V4L2_SEL_TGT_COMPOSE``
> > +rectangle, relative to the analogue crop rectangle, on (pad, stream) pair
> > +1/0. It depends on the driver which of these operations are being used to
> > +achieve the resulting size.
>
> I would say "It depends on the driver implementation to decide how
> binning and skipping are combined to obtain the desired scaling
> ratio."
I wouldn't refer to this as scaling as we'll have specific scaling
configuration elsewhere. How about:
The driver implementation determines how to configure binning and
sub-sampling to achieve the desired size.
>
> > +
> > +The digital crop operation takes place after binning and sub-sampling. It is
> > +configured by setting the ``V4L2_SEL_TGT_CROP`` rectangle on (pad, stream) pair
> > +0/0. The resulting image size is further output by the sensor.
>
> This is not true anymore if we consider digital scaling and
> post-scaler crop.
Yes. I'll address this in the next patch.
>
> > +
> > +The sensor's output mbus code is configured by setting the format on the (pad,
> > +stream) pair 0/0. When setting the format, always use the same width and height
> > +as for the digital crop setting.
> > +
>
> Same comment here as well.
Ditto.
>
> > +Drivers may only support some of even none of these configurations, in which
>
> s/configurations/configuration options/
>
> > +case they do not expose the corresponding selection rectangles. If any selection
>
> How are selection rectangles "exposed" ?
Well, they are not imposed at least. :-)
> I would say "support" instead of expose.
>
> However I'm a bit concerned what "support" (or "expose") means. We
> should probably define it. I do think that driver should allow reading
> all the above selection rectangles and return the correct values (if a
> configuration step is not supported, sizes are simply passed forward
> from the previous selection target to the next target/format). I'm
> less sure about setting them, if they should error out or simply
> adjust it as get_selection() would have done.
Either could be used IMO. If something is not supported and the user tries
to access it, the result is an error (-EINVAL in this case).
>
>
> > +targets are omitted, the further selection rectangle or format is instead
> > +related to the previous implemented selection rectangle. For instance, if the
> > +sensor supports binning but not analogue crop, then the binning configuration
> > +(``V4L2_SEL_TGT_COMPOSE`` selection target) is done in relation to the visible
> > +pixel area (``V4L2_SEL_TGT_CROP_DEFAULT`` selection target).
>
> Let alone the fact I would have used, say, digital crop as an example
> of an optional feature, I think allowing to read all the possible
> targets would save userspace keeping track of what target the
> rectangle they want to configure refers to.
This is how the selection API works. If we want to deviate from that, it's
another thing, but currently if a driver doesn't support configuring
a selection, it won't support that target either.
If we required all selection rectangles to be supported even if they
wouldn't be configurable, then it'd be up to user to try to change them to
see if they can be modified. I'm not sure if that would be an improvement
as a whole.
I wonder what Laurent thinks.
>
> > +
> > +Also refer to :ref:`Selection targets <v4l2-selection-targets-table>`.
> > +
> > +.. flat-table:: Selection targets on pads
> > + :header-rows: 1
> > +
> > + * - Pad/Stream
> > + - Selection target/format
> > + - Mandatory (X/-)
> > + - Modifiable (X/-)
> > + - Synopsis
> > + * - 1/0
> > + - Format
> > + - X
> > + - \-
> > + - Image data format. The width and the height fields indicates the full
> > + size of the pixel array, including non-visible pixels. The media bus
> > + code of this format reflects the native pixel depth of the sensor.
> > + * - 1/0
> > + - ``V4L2_SEL_TGT_CROP_DEFAULT``
> > + - X
> > + - \
> > + - The visible pixel area. This rectangle is relative to the format on the
> > + same (pad, stream).
> > + * - 1/0
> > + - ``V4L2_SEL_TGT_CROP``
> > + - \-
> > + - X
> > + - Analogue crop. Analogue crop typically has a coarse granularity. This
> > + rectangle is relative to the format on the same (pad, stream).
> > + * - 1/0
> > + - ``V4L2_SEL_TGT_COMPOSE``
> > + - \-
> > + - X
> > + - Binning and sub-sampling. This rectangle is relative to the
> > + ``V4L2_SEL_TGT_CROP`` rectangle on the same (pad, stream). The
> > + combination of binning and sub-sampling is configured using this
> > + selection target.
> > + * - 2/0
> > + - Format
> > + - X
> > + - \-
> > + - Embedded data format.
> > + * - 0/0
> > + - ``V4L2_SEL_TGT_CROP``
> > + - \-
> > + - X
> > + - Digital crop. This rectangle is relative to the ``V4L2_SEL_TGT_COMPOSE``
> > + rectangle on (pad, stream) pair 1/0.
> > + * - 0/0
> > + - Format
> > + - X
> > + - X
> > + - Image data source format. Always assign the width and height fields of
> > + the format to the same values than for the ``V4L2_SEL_TGT_CROP``
> > + rectangle on (pad, stream) pair 0/0. The media bus code reflects the
> > + pixel data output of the sensor.
>
> This makes me think it is intentional not to document digital
> scaling/post-scaler crop in this version ?
Not in this patch, no. :-)
I'm fine merging this to the 5th patch if there's an agreement they should
be merged together (probably?).
>
> > + * - 0/1
> > + - Format
> > + - X
> > + - \-
> > + - Embedded data source format.
> > +
> > +Embedded data
> > +^^^^^^^^^^^^^
> > +
> > +The embedded data stream is produced by the sensor when the corresponding route
> > +is enabled. The embedded data route may also be immutable or not exist at all,
> > +in case the sensor (or the driver) does not support it.
> > +
> > +Generally the sensor embedded data width is determined by the width of the image
> > +data whereas the number of lines are constant for the embedded data. The user
>
> Maybe drop "The" from "The user space" ?
I'd keep it as-is.
>
> > +space may obtain the size of the embedded data once the image data size on the
> > +source pad has been configured.
> > +
> > +Also see :ref:`media_using_camera_sensor_drivers_embedded_data`.
>
> Thanks for the hard work here, I like how this new document
> clearly outlines the expected interfaces exposed by drivers.
Thank you. Hopefully we could all agree on this soonish and get libcamera
support implemented so we could merge it all. :-)
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 4/9] media: Documentation: Add subdev configuration models, raw sensor model
2024-12-04 13:22 ` Sakari Ailus
@ 2024-12-04 13:37 ` Jacopo Mondi
2024-12-04 14:20 ` Sakari Ailus
0 siblings, 1 reply; 25+ messages in thread
From: Jacopo Mondi @ 2024-12-04 13:37 UTC (permalink / raw)
To: Sakari Ailus
Cc: Jacopo Mondi, linux-media, hverkuil, laurent.pinchart, Prabhakar,
Kate Hsuan, Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Sakari
On Wed, Dec 04, 2024 at 01:22:30PM +0000, Sakari Ailus wrote:
> Hi Jacopo,
>
> Thank you for the review.
>
> On Wed, Dec 04, 2024 at 01:10:55PM +0100, Jacopo Mondi wrote:
> > Hi Sakari
> >
> > On Fri, Nov 29, 2024 at 11:51:37AM +0200, Sakari Ailus wrote:
> > > Sub-device configuration models define what V4L2 API elements are
> > > available on a compliant sub-device and how do they behave.
> > >
> > > The patch also adds a model for common raw sensors.
> > >
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > > .../media/drivers/camera-sensor.rst | 4 +
> > > .../media/v4l/common-raw-sensor.dia | 441 ++++++++++++++++++
> > > .../media/v4l/common-raw-sensor.svg | 134 ++++++
> > > .../userspace-api/media/v4l/dev-subdev.rst | 2 +
> > > .../media/v4l/subdev-config-model.rst | 208 +++++++++
> > > 5 files changed, 789 insertions(+)
> > > create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.dia
> > > create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.svg
> > > create mode 100644 Documentation/userspace-api/media/v4l/subdev-config-model.rst
> > >
> > > diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > > index bc55c861fb69..5bc4c79d230c 100644
> > > --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > > +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > > @@ -18,6 +18,8 @@ binning functionality. The sensor drivers belong to two distinct classes, freely
> > > configurable and register list-based drivers, depending on how the driver
> > > configures this functionality.
> > >
> > > +Also see :ref:`media_subdev_config_model_common_raw_sensor`.
> > > +
> > > Freely configurable camera sensor drivers
> > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > >
> > > @@ -105,6 +107,8 @@ values programmed by the register sequences. The default values of these
> > > controls shall be 0 (disabled). Especially these controls shall not be inverted,
> > > independently of the sensor's mounting rotation.
> > >
> > > +.. _media_using_camera_sensor_drivers_embedded_data:
> > > +
> > > Embedded data
> > > -------------
> > >
> > > diff --git a/Documentation/userspace-api/media/v4l/common-raw-sensor.dia b/Documentation/userspace-api/media/v4l/common-raw-sensor.dia
> >
> > snip
> >
> > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > index dcfcbd52490d..4d145bd3bd09 100644
> > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > @@ -838,3 +838,5 @@ stream while it may be possible to enable and disable the embedded data stream.
> > >
> > > The embedded data format does not need to be configured on the sensor's pads as
> > > the format is dictated by the pixel data format in this case.
> > > +
> > > +.. include:: subdev-config-model.rst
> > > diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> > > new file mode 100644
> > > index 000000000000..4ddf98e3143c
> > > --- /dev/null
> > > +++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> > > @@ -0,0 +1,208 @@
> > > +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> > > +
> > > +.. _media_subdev_config_model:
> > > +
> > > +Sub-device configuration models
> > > +===============================
> > > +
> > > +A sub-device configuration model specifies in detail what the user space can
> > > +expect from a sub-device in terms of V4L2 sub-device interface support,
> > > +including IOCTL (including selection targets and controls) semantics.
> >
> > including IOCTLs, selection targets and controls semantics.
>
> I think we could as well as drop these, i.e. just mention semantics.
>
ack
> >
> > > +
> > > +A sub-device may implement more than one configuration model at the same
> > > +time. The implemented configuration models can be obtained from the sub-device's
> > > +``V4L2_CID_CONFIG_MODEL`` control.
> > > +
> > > +.. _media_subdev_config_model_common_raw_sensor:
> > > +
> > > +Common raw camera sensor model
> > > +------------------------------
> > > +
> > > +The common raw camera sensor model defines a set of enumeration and
> > > +configuration interfaces (formats, selections etc.) that cover the vast majority
> > > +of funcitionality of raw camera sensors. Not all of the interfaces are
> >
> > s/funcitionality/functionalities
>
> I'd say singular is right in this case. Maybe Kieran or Dave have an
> opinion as well? :-)
Then
s/funcitionality/functionality
:)
>
> >
> > > +necessarily offered by all drivers.
> > > +
> > > +A sub-device complies with the common raw sensor model if the
> > > +``V4L2_CONFIG_MODEL_COMMON_RAW`` bit is set in the ``V4L2_CID_CONFIG_MODEL``
> > > +control of the sub-device.
> > > +
> > > +The common raw camera sensor model is aligned with
> > > +:ref:`media_using_camera_sensor_drivers`. Please refer to that regarding aspects
> > > +not specified here.
> > > +
> > > +Each camera sensor implementing the common raw sensor model exposes a single
> > > +V4L2 sub-device. The sub-device contains a single source pad (0) and two or more
> > > +internal pads: an image data internal pad (1) and optionally an embedded data
> > > +pad (2). Additionally, further internal pads may be supported for other
> > > +features, in which case they are documented separately for the given device.
> > > +
> > > +This is shown in :ref:`media_subdev_config_model_common_raw_sensor_subdev`.
> > > +
> > > +.. _media_subdev_config_model_common_raw_sensor_subdev:
> > > +
> > > +.. kernel-figure:: common-raw-sensor.svg
> > > + :alt: common-raw-sensor.svg
> > > + :align: center
> > > +
> > > + **Common raw sensor sub-device**
> > > +
> > > +Routes
> > > +^^^^^^
> > > +
> > > +A sub-device conforming to common raw camera sensor model implements the
> > > +following routes.
> > > +
> > > +.. flat-table:: Routes
> > > + :header-rows: 1
> > > +
> > > + * - Sink pad/stream
> > > + - Source pad/stream
> > > + - Static (X/M(aybe)/-)
> > > + - Mandatory (X/-)
> > > + - Synopsis
> > > + * - 1/0
> > > + - 0/0
> > > + - X
> > > + - X
> > > + - Image data
> > > + * - 2/0
> > > + - 0/1
> > > + - M
> > > + - \-
> > > + - Embedded data
> > > +
> > > +Some devices do not support the embedded data stream, others do support it and
> > > +in some of the latter, it can be turned on and off before streaming is started.
> >
> > What about:
> >
> > Support for the embedded data stream is optional. Device that support
> > the embedded data stream might allow to enable/disable the route
> > before the streaming is started.
>
> How about:
>
> Support for the embedded data stream is optional. Drivers supporting
> the embedded data stream may allow disabling and enabling the route
> when the streaming is disabled.
>
Yeah, better
>
> >
> > > +
> > > +Sensor pixel array size, cropping and binning
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +
> > > +The sensor's pixel array is divided into one or more areas. The areas around the
> > > +edge of the pixel array, usually one or more sides, may contain optical black
> >
> > optically ?
>
> E.g. CCS spec refers to these as "optical black" pixels.
>
ack then
> >
> > > +pixels, dummy pixels and other non-image pixels. The entire pixel array size is
> > > +conveyed by the format on (pad, stream) pair 1/0.
> > > +
> > > +A rectangle within the pixel array contains the visible pixels. Capturing the
> > > +non-visible pixels outside the visible pixel area may be supported by the
> > > +sensor. The visible pixel area corresponds to the ``V4L2_SEL_TGT_CROP_DEFAULT``
> > > +selection target on (pad, stream) pair 1/0.
> > > +
> > > +Sensors can perform multiple operations that affect the output image size. First
> > > +of these is the analogue crop. Analogue crop limits the area of the pixel array
> > > +which the sensor will read, affecting sensor timing as well. The granularity of
> > > +the analogue crop configuration varies greatly across sensors: some sensors
> > > +support only a few different analogue crop configurations whereas others may
> > > +support anything divisible by a given number of pixels. The analogue crop
> > > +configuration corresponds the ``V4L2_SEL_TGT_CROP`` selection target on (pad,
> > > +stream) pair 1/0. The default analogue crop rectangle corresponds to the visible
> > > +pixel area.
> > > +
> > > +In the next step, binning is performed on the image data read from camera
> > > +sensor's pixel array, as determined by the analogue crop configuration. Enabling
> > > +binning will effectively result in an image smaller than the original by given
> > > +binning factors horizontally and vertically. Typical values are 1/2 and 1/3 but
> >
> > I thought 1/4 was more common than 1/3... Nevermind..
> >
> > > +others may well be supported by the hardware as well.
> > > +
> > > +Sub-sampling follows binning. Sub-sampling, like binning, reduces the size of
> > > +the image by including only a subset of samples read from the sensor's pixel
> > > +matrix, typically every n'th pixel horizontally and vertically, taking the
> > > +sensor's colour pattern into account. Sub-sampling is generally configurable
> > > +separately horizontally and vertically.
> > > +
> > > +Binning and sub-sampling are configured using the ``V4L2_SEL_TGT_COMPOSE``
> > > +rectangle, relative to the analogue crop rectangle, on (pad, stream) pair
> > > +1/0. It depends on the driver which of these operations are being used to
> > > +achieve the resulting size.
> >
> > I would say "It depends on the driver implementation to decide how
> > binning and skipping are combined to obtain the desired scaling
> > ratio."
>
> I wouldn't refer to this as scaling as we'll have specific scaling
> configuration elsewhere. How about:
>
> The driver implementation determines how to configure binning and
> sub-sampling to achieve the desired size.
>
Fine with me indeed
> >
> > > +
> > > +The digital crop operation takes place after binning and sub-sampling. It is
> > > +configured by setting the ``V4L2_SEL_TGT_CROP`` rectangle on (pad, stream) pair
> > > +0/0. The resulting image size is further output by the sensor.
> >
> > This is not true anymore if we consider digital scaling and
> > post-scaler crop.
>
> Yes. I'll address this in the next patch.
>
> >
> > > +
> > > +The sensor's output mbus code is configured by setting the format on the (pad,
> > > +stream) pair 0/0. When setting the format, always use the same width and height
> > > +as for the digital crop setting.
> > > +
> >
> > Same comment here as well.
>
> Ditto.
>
> >
> > > +Drivers may only support some of even none of these configurations, in which
> >
> > s/configurations/configuration options/
> >
> > > +case they do not expose the corresponding selection rectangles. If any selection
> >
> > How are selection rectangles "exposed" ?
>
> Well, they are not imposed at least. :-)
>
> > I would say "support" instead of expose.
> >
> > However I'm a bit concerned what "support" (or "expose") means. We
> > should probably define it. I do think that driver should allow reading
> > all the above selection rectangles and return the correct values (if a
> > configuration step is not supported, sizes are simply passed forward
> > from the previous selection target to the next target/format). I'm
> > less sure about setting them, if they should error out or simply
> > adjust it as get_selection() would have done.
>
> Either could be used IMO. If something is not supported and the user tries
> to access it, the result is an error (-EINVAL in this case).
>
> >
> >
> > > +targets are omitted, the further selection rectangle or format is instead
> > > +related to the previous implemented selection rectangle. For instance, if the
> > > +sensor supports binning but not analogue crop, then the binning configuration
> > > +(``V4L2_SEL_TGT_COMPOSE`` selection target) is done in relation to the visible
> > > +pixel area (``V4L2_SEL_TGT_CROP_DEFAULT`` selection target).
> >
> > Let alone the fact I would have used, say, digital crop as an example
> > of an optional feature, I think allowing to read all the possible
> > targets would save userspace keeping track of what target the
> > rectangle they want to configure refers to.
>
> This is how the selection API works. If we want to deviate from that, it's
> another thing, but currently if a driver doesn't support configuring
> a selection, it won't support that target either.
>
> If we required all selection rectangles to be supported even if they
> wouldn't be configurable, then it'd be up to user to try to change them to
> see if they can be modified. I'm not sure if that would be an improvement
> as a whole.
>
Yeah, probably a set-then-verify-if-changed is more cumbersome than
detecting an -EINVAL on set_selection.
I'm fine with this as long as all the mentioned targets are readable.
> I wonder what Laurent thinks.
>
> >
> > > +
> > > +Also refer to :ref:`Selection targets <v4l2-selection-targets-table>`.
> > > +
> > > +.. flat-table:: Selection targets on pads
> > > + :header-rows: 1
> > > +
> > > + * - Pad/Stream
> > > + - Selection target/format
> > > + - Mandatory (X/-)
> > > + - Modifiable (X/-)
> > > + - Synopsis
> > > + * - 1/0
> > > + - Format
> > > + - X
> > > + - \-
> > > + - Image data format. The width and the height fields indicates the full
> > > + size of the pixel array, including non-visible pixels. The media bus
> > > + code of this format reflects the native pixel depth of the sensor.
> > > + * - 1/0
> > > + - ``V4L2_SEL_TGT_CROP_DEFAULT``
> > > + - X
> > > + - \
> > > + - The visible pixel area. This rectangle is relative to the format on the
> > > + same (pad, stream).
> > > + * - 1/0
> > > + - ``V4L2_SEL_TGT_CROP``
> > > + - \-
> > > + - X
> > > + - Analogue crop. Analogue crop typically has a coarse granularity. This
> > > + rectangle is relative to the format on the same (pad, stream).
> > > + * - 1/0
> > > + - ``V4L2_SEL_TGT_COMPOSE``
> > > + - \-
> > > + - X
> > > + - Binning and sub-sampling. This rectangle is relative to the
> > > + ``V4L2_SEL_TGT_CROP`` rectangle on the same (pad, stream). The
> > > + combination of binning and sub-sampling is configured using this
> > > + selection target.
> > > + * - 2/0
> > > + - Format
> > > + - X
> > > + - \-
> > > + - Embedded data format.
> > > + * - 0/0
> > > + - ``V4L2_SEL_TGT_CROP``
> > > + - \-
> > > + - X
> > > + - Digital crop. This rectangle is relative to the ``V4L2_SEL_TGT_COMPOSE``
> > > + rectangle on (pad, stream) pair 1/0.
> > > + * - 0/0
> > > + - Format
> > > + - X
> > > + - X
> > > + - Image data source format. Always assign the width and height fields of
> > > + the format to the same values than for the ``V4L2_SEL_TGT_CROP``
> > > + rectangle on (pad, stream) pair 0/0. The media bus code reflects the
> > > + pixel data output of the sensor.
> >
> > This makes me think it is intentional not to document digital
> > scaling/post-scaler crop in this version ?
>
> Not in this patch, no. :-)
>
> I'm fine merging this to the 5th patch if there's an agreement they should
> be merged together (probably?).
>
I would probably merge the two, yes
> >
> > > + * - 0/1
> > > + - Format
> > > + - X
> > > + - \-
> > > + - Embedded data source format.
> > > +
> > > +Embedded data
> > > +^^^^^^^^^^^^^
> > > +
> > > +The embedded data stream is produced by the sensor when the corresponding route
> > > +is enabled. The embedded data route may also be immutable or not exist at all,
> > > +in case the sensor (or the driver) does not support it.
> > > +
> > > +Generally the sensor embedded data width is determined by the width of the image
> > > +data whereas the number of lines are constant for the embedded data. The user
> >
> > Maybe drop "The" from "The user space" ?
>
> I'd keep it as-is.
>
> >
> > > +space may obtain the size of the embedded data once the image data size on the
> > > +source pad has been configured.
> > > +
> > > +Also see :ref:`media_using_camera_sensor_drivers_embedded_data`.
> >
> > Thanks for the hard work here, I like how this new document
> > clearly outlines the expected interfaces exposed by drivers.
>
> Thank you. Hopefully we could all agree on this soonish and get libcamera
> support implemented so we could merge it all. :-)
>
Yep, very close now!
Thanks
j
> --
> Kind regards,
>
> Sakari Ailus
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 5/9] media: Documentation: Add scaling and post-scaler crop for common raw
2024-12-04 11:25 ` Jacopo Mondi
@ 2024-12-04 14:15 ` Sakari Ailus
2024-12-04 15:33 ` Jacopo Mondi
0 siblings, 1 reply; 25+ messages in thread
From: Sakari Ailus @ 2024-12-04 14:15 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Jacopo,
Thank you for the review.
On Wed, Dec 04, 2024 at 12:25:11PM +0100, Jacopo Mondi wrote:
> Hi Sakari
> thanks for the update
>
> On Fri, Nov 29, 2024 at 11:51:38AM +0200, Sakari Ailus wrote:
> > Document scaling and post-scaler digital crop operations for the common
> > raw sensor model.
> > Signedg-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > ---
> > .../media/v4l/subdev-config-model.rst | 19 +++++++++++++++----
> > 1 file changed, 15 insertions(+), 4 deletions(-)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> > index 4ddf98e3143c..1ae20800f34b 100644
> > --- a/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> > +++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> > @@ -119,9 +119,13 @@ The digital crop operation takes place after binning and sub-sampling. It is
> > configured by setting the ``V4L2_SEL_TGT_CROP`` rectangle on (pad, stream) pair
> > 0/0. The resulting image size is further output by the sensor.
>
> I think this last line "The resulting image size is further output by
> the sensor." doesn't apply anymore now that we have [digital crop +
> digital scaling + post-scaling crop].
Indeed. I'll address this in v4.
>
> >
> > +The digital scaling operation is performed after the digital crop. It is
> > +configured by setting the ``V4L2_SEL_TGT_COMPOSE`` rectangle on (pad, stream) pair
> > +0/0, relative to the digital crop.
> > +
>
> "to the digital crop rectangle." maybe ?
Yes.
>
>
> > The sensor's output mbus code is configured by setting the format on the (pad,
> > -stream) pair 0/0. When setting the format, always use the same width and height
> > -as for the digital crop setting.
> > +stream) pair 0/0. The width and height fields are used to configure post-scaler
> > +digital crop, affecting the right side and the bottom of the frame.
> >
>
> I would introduce the (optional) presence of an additional post-scaler
> digital crop step explicitly, also I get this is kind of rare feature,
> isn't it ?
>
> -------------------------------------------------------------------------------
> The digital scaling operation is performed after the digital crop. It
> is configured by setting the ``V4L2_SEL_TGT_COMPOSE`` rectangle on
> (pad, stream) pair 0/0, relative to the digital crop rectangle.
>
> The sensor's output mbus code is configured by setting the format on the (pad,
> stream) pair 0/0. An optional post-scaler crop step might be performed by
> specifying a width and height smaller than the digital scaling
> rectangle. If post-scaler cropping is not supported the format's sizes
> will always match the compose rectangle sizes applied on the same 0/0
> (pad, stream) pair.
Right, even if the post-scaler crop is optional, the format is always
there but unmodifiable (directly). I'd refer to driver support instead of
somewhat nebulous "might be performed".
How about:
The sensor's output mbus code is configured by setting the format on the
(pad, stream) pair 0/0. The width and height fields are used to configure
post-scaler digital crop if supported by the driver, affecting the right
and bottom edges of the frame. If post-scaler digital crop is not
supported, the width and height fields of the format will match the compose
rectangle sizes applied on the same 0/0 (pad, stream) pair.
> -------------------------------------------------------------------------------
>
> I didn't get when we spoke about it where the "affecting the right
> side and bottom of the frame" comes from when talking about
> post-scaler cropping.
>
>
> > Drivers may only support some of even none of these configurations, in which
> > case they do not expose the corresponding selection rectangles. If any selection
> > @@ -179,12 +183,19 @@ Also refer to :ref:`Selection targets <v4l2-selection-targets-table>`.
> > - X
> > - Digital crop. This rectangle is relative to the ``V4L2_SEL_TGT_COMPOSE``
> > rectangle on (pad, stream) pair 1/0.
> > + * - 0/0
> > + - ``V4L2_SEL_TGT_COMPOSE``
> > + - \-
> > + - X
> > + - Scaling. This rectangle is relative to the ``V4L2_SEL_TGT_CROP``
>
> "Digital scaling" ? or is not necessary ?
We don't mention it in the context of sub-sampling either. Digital and
analogue variants are separately documented where they exist, I think this
should be fine. What do you think?
>
> > + rectangle on (pad, stream) pair 0/0.
> > * - 0/0
> > - Format
> > - X
> > - X
> > - - Image data source format. Always assign the width and height fields of
> > - the format to the same values than for the ``V4L2_SEL_TGT_CROP``
> > + - Image data source format and post-scaler crop. The width and height
> > + fields of the format, used to configure post-scaler crop on the right
> > + and bottom edges of the image, are related to the ``V4L2_SEL_TGT_COMPOSE``
> > rectangle on (pad, stream) pair 0/0. The media bus code reflects the
> > pixel data output of the sensor.
> > * - 0/1
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 4/9] media: Documentation: Add subdev configuration models, raw sensor model
2024-12-04 13:37 ` Jacopo Mondi
@ 2024-12-04 14:20 ` Sakari Ailus
0 siblings, 0 replies; 25+ messages in thread
From: Sakari Ailus @ 2024-12-04 14:20 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Jacopo,
On Wed, Dec 04, 2024 at 02:37:21PM +0100, Jacopo Mondi wrote:
> > > > +Common raw camera sensor model
> > > > +------------------------------
> > > > +
> > > > +The common raw camera sensor model defines a set of enumeration and
> > > > +configuration interfaces (formats, selections etc.) that cover the vast majority
> > > > +of funcitionality of raw camera sensors. Not all of the interfaces are
> > >
> > > s/funcitionality/functionalities
> >
> > I'd say singular is right in this case. Maybe Kieran or Dave have an
> > opinion as well? :-)
>
>
> Then
> s/funcitionality/functionality
>
> :)
Ah, indeed. ;-)
...
> > > > +targets are omitted, the further selection rectangle or format is instead
> > > > +related to the previous implemented selection rectangle. For instance, if the
> > > > +sensor supports binning but not analogue crop, then the binning configuration
> > > > +(``V4L2_SEL_TGT_COMPOSE`` selection target) is done in relation to the visible
> > > > +pixel area (``V4L2_SEL_TGT_CROP_DEFAULT`` selection target).
> > >
> > > Let alone the fact I would have used, say, digital crop as an example
> > > of an optional feature, I think allowing to read all the possible
> > > targets would save userspace keeping track of what target the
> > > rectangle they want to configure refers to.
> >
> > This is how the selection API works. If we want to deviate from that, it's
> > another thing, but currently if a driver doesn't support configuring
> > a selection, it won't support that target either.
> >
> > If we required all selection rectangles to be supported even if they
> > wouldn't be configurable, then it'd be up to user to try to change them to
> > see if they can be modified. I'm not sure if that would be an improvement
> > as a whole.
> >
>
> Yeah, probably a set-then-verify-if-changed is more cumbersome than
> detecting an -EINVAL on set_selection.
>
> I'm fine with this as long as all the mentioned targets are readable.
Currently it's how this is documented in V4L2 sub-device UAPI
documentation: if you have nothing to configure, then there's no selection
target either.
> > > This makes me think it is intentional not to document digital
> > > scaling/post-scaler crop in this version ?
> >
> > Not in this patch, no. :-)
> >
> > I'm fine merging this to the 5th patch if there's an agreement they should
> > be merged together (probably?).
> >
>
> I would probably merge the two, yes
Ack. Let's wait for the discussion to conclude.
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 7/9] media: uapi: Add V4L2_CID_BINNING control for binning configuration
2024-12-04 11:33 ` Jacopo Mondi
@ 2024-12-04 14:24 ` Sakari Ailus
0 siblings, 0 replies; 25+ messages in thread
From: Sakari Ailus @ 2024-12-04 14:24 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Jacopo,
On Wed, Dec 04, 2024 at 12:33:28PM +0100, Jacopo Mondi wrote:
> Hi Sakari
>
> On Fri, Nov 29, 2024 at 11:51:40AM +0200, Sakari Ailus wrote:
> > Add V4L2_CID_BINNING control for configuting binning and enumerating a
> > camera sensor's binning capabilities. The control combines horizontal and
> > vertical binning into a single control as the two are generally related.
> >
> > New drivers should use this control to configure binning.
>
> I thought the idea was to post-pone this (and the following 2 patches)
> to when controls will be able to be tested without changing the device
> configuration, to properly support TRY_FORMAT, and before that time
> work with selection rectangles and let the drivers select the
> "correct" binning/scaling factors deduced from the
> TGT_CROP-to-TGT_COMPOSE ratio on (pad, stream) 1/0
As I wrote on the cover letter, some of the patches in the set will likely
be postponed. Most likely this will mean the binning and sub-sampling
control patches. I still wanted to include them so we'd give some view on
how this would look like once we have control values in sub-device state.
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 5/9] media: Documentation: Add scaling and post-scaler crop for common raw
2024-12-04 14:15 ` Sakari Ailus
@ 2024-12-04 15:33 ` Jacopo Mondi
2024-12-05 6:56 ` Sakari Ailus
0 siblings, 1 reply; 25+ messages in thread
From: Jacopo Mondi @ 2024-12-04 15:33 UTC (permalink / raw)
To: Sakari Ailus
Cc: Jacopo Mondi, linux-media, hverkuil, laurent.pinchart, Prabhakar,
Kate Hsuan, Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Sakari
On Wed, Dec 04, 2024 at 02:15:07PM +0000, Sakari Ailus wrote:
> Hi Jacopo,
>
> Thank you for the review.
>
> On Wed, Dec 04, 2024 at 12:25:11PM +0100, Jacopo Mondi wrote:
> > Hi Sakari
> > thanks for the update
> >
> > On Fri, Nov 29, 2024 at 11:51:38AM +0200, Sakari Ailus wrote:
> > > Document scaling and post-scaler digital crop operations for the common
> > > raw sensor model.
> > > Signedg-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > ---
> > > .../media/v4l/subdev-config-model.rst | 19 +++++++++++++++----
> > > 1 file changed, 15 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> > > index 4ddf98e3143c..1ae20800f34b 100644
> > > --- a/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> > > +++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> > > @@ -119,9 +119,13 @@ The digital crop operation takes place after binning and sub-sampling. It is
> > > configured by setting the ``V4L2_SEL_TGT_CROP`` rectangle on (pad, stream) pair
> > > 0/0. The resulting image size is further output by the sensor.
> >
> > I think this last line "The resulting image size is further output by
> > the sensor." doesn't apply anymore now that we have [digital crop +
> > digital scaling + post-scaling crop].
>
> Indeed. I'll address this in v4.
>
> >
> > >
> > > +The digital scaling operation is performed after the digital crop. It is
> > > +configured by setting the ``V4L2_SEL_TGT_COMPOSE`` rectangle on (pad, stream) pair
> > > +0/0, relative to the digital crop.
> > > +
> >
> > "to the digital crop rectangle." maybe ?
>
> Yes.
>
> >
> >
> > > The sensor's output mbus code is configured by setting the format on the (pad,
> > > -stream) pair 0/0. When setting the format, always use the same width and height
> > > -as for the digital crop setting.
> > > +stream) pair 0/0. The width and height fields are used to configure post-scaler
> > > +digital crop, affecting the right side and the bottom of the frame.
> > >
> >
> > I would introduce the (optional) presence of an additional post-scaler
> > digital crop step explicitly, also I get this is kind of rare feature,
> > isn't it ?
> >
> > -------------------------------------------------------------------------------
> > The digital scaling operation is performed after the digital crop. It
> > is configured by setting the ``V4L2_SEL_TGT_COMPOSE`` rectangle on
> > (pad, stream) pair 0/0, relative to the digital crop rectangle.
> >
> > The sensor's output mbus code is configured by setting the format on the (pad,
> > stream) pair 0/0. An optional post-scaler crop step might be performed by
> > specifying a width and height smaller than the digital scaling
> > rectangle. If post-scaler cropping is not supported the format's sizes
> > will always match the compose rectangle sizes applied on the same 0/0
> > (pad, stream) pair.
>
> Right, even if the post-scaler crop is optional, the format is always
> there but unmodifiable (directly). I'd refer to driver support instead of
> somewhat nebulous "might be performed".
>
> How about:
>
> The sensor's output mbus code is configured by setting the format on the
> (pad, stream) pair 0/0. The width and height fields are used to configure
> post-scaler digital crop if supported by the driver, affecting the right
> and bottom edges of the frame. If post-scaler digital crop is not
> supported, the width and height fields of the format will match the compose
> rectangle sizes applied on the same 0/0 (pad, stream) pair.
>
Fine indeed!
> > -------------------------------------------------------------------------------
> >
> > I didn't get when we spoke about it where the "affecting the right
> > side and bottom of the frame" comes from when talking about
> > post-scaler cropping.
> >
> >
> > > Drivers may only support some of even none of these configurations, in which
> > > case they do not expose the corresponding selection rectangles. If any selection
> > > @@ -179,12 +183,19 @@ Also refer to :ref:`Selection targets <v4l2-selection-targets-table>`.
> > > - X
> > > - Digital crop. This rectangle is relative to the ``V4L2_SEL_TGT_COMPOSE``
> > > rectangle on (pad, stream) pair 1/0.
> > > + * - 0/0
> > > + - ``V4L2_SEL_TGT_COMPOSE``
> > > + - \-
> > > + - X
> > > + - Scaling. This rectangle is relative to the ``V4L2_SEL_TGT_CROP``
> >
> > "Digital scaling" ? or is not necessary ?
>
> We don't mention it in the context of sub-sampling either. Digital and
> analogue variants are separately documented where they exist, I think this
> should be fine. What do you think?
>
I'm fine, after all there's no "analogue scaling" this can be confused
with, right ?
> >
> > > + rectangle on (pad, stream) pair 0/0.
> > > * - 0/0
> > > - Format
> > > - X
> > > - X
> > > - - Image data source format. Always assign the width and height fields of
> > > - the format to the same values than for the ``V4L2_SEL_TGT_CROP``
> > > + - Image data source format and post-scaler crop. The width and height
> > > + fields of the format, used to configure post-scaler crop on the right
> > > + and bottom edges of the image, are related to the ``V4L2_SEL_TGT_COMPOSE``
> > > rectangle on (pad, stream) pair 0/0. The media bus code reflects the
> > > pixel data output of the sensor.
> > > * - 0/1
>
> --
> Kind regards,
>
> Sakari Ailus
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 5/9] media: Documentation: Add scaling and post-scaler crop for common raw
2024-12-04 15:33 ` Jacopo Mondi
@ 2024-12-05 6:56 ` Sakari Ailus
0 siblings, 0 replies; 25+ messages in thread
From: Sakari Ailus @ 2024-12-05 6:56 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Jacopo,
On Wed, Dec 04, 2024 at 04:33:22PM +0100, Jacopo Mondi wrote:
> > > > @@ -179,12 +183,19 @@ Also refer to :ref:`Selection targets <v4l2-selection-targets-table>`.
> > > > - X
> > > > - Digital crop. This rectangle is relative to the ``V4L2_SEL_TGT_COMPOSE``
> > > > rectangle on (pad, stream) pair 1/0.
> > > > + * - 0/0
> > > > + - ``V4L2_SEL_TGT_COMPOSE``
> > > > + - \-
> > > > + - X
> > > > + - Scaling. This rectangle is relative to the ``V4L2_SEL_TGT_CROP``
> > >
> > > "Digital scaling" ? or is not necessary ?
> >
> > We don't mention it in the context of sub-sampling either. Digital and
> > analogue variants are separately documented where they exist, I think this
> > should be fine. What do you think?
> >
>
> I'm fine, after all there's no "analogue scaling" this can be confused
> with, right ?
Correct.
--
Sakari Ailus
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 2/9] media: Documentation: Add a hyphen to list-based
2024-11-29 9:51 ` [RFC v3 2/9] media: Documentation: Add a hyphen to list-based Sakari Ailus
@ 2024-12-05 8:04 ` Jacopo Mondi
0 siblings, 0 replies; 25+ messages in thread
From: Jacopo Mondi @ 2024-12-05 8:04 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Sakari
On Fri, Nov 29, 2024 at 11:51:35AM +0200, Sakari Ailus wrote:
> Add a hyphen to list-based for uniform spelling in camera-sensor.rst.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
> Documentation/userspace-api/media/drivers/camera-sensor.rst | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> index 8e1083417ae1..9a1e8aa9fc77 100644
> --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> @@ -26,10 +26,10 @@ of cropping and scaling operations from the device's pixel array's size.
>
> An example of such a driver is the CCS driver.
>
> -Register list based drivers
> +Register list-based drivers
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> -Register list based drivers generally, instead of able to configure the device
> +Register list-based drivers generally, instead of able to configure the device
> they control based on user requests, are limited to a number of preset
> configurations that combine a number of different parameters that on hardware
> level are independent. How a driver picks such configuration is based on the
> @@ -67,7 +67,7 @@ is pixels and the unit of the ``V4L2_CID_VBLANK`` is lines. The pixel rate in
> the sensor's **pixel array** is specified by ``V4L2_CID_PIXEL_RATE`` in the same
> sub-device. The unit of that control is pixels per second.
>
> -Register list based drivers need to implement read-only sub-device nodes for the
> +Register list-based drivers need to implement read-only sub-device nodes for the
> purpose. Devices that are not register list based need these to configure the
> device's internal processing pipeline.
>
> --
> 2.39.5
>
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 3/9] media: Documentation: Reword split of sensor driver to two classes
2024-11-29 9:51 ` [RFC v3 3/9] media: Documentation: Reword split of sensor driver to two classes Sakari Ailus
@ 2024-12-05 8:06 ` Jacopo Mondi
0 siblings, 0 replies; 25+ messages in thread
From: Jacopo Mondi @ 2024-12-05 8:06 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Sakari
On Fri, Nov 29, 2024 at 11:51:36AM +0200, Sakari Ailus wrote:
> The sensor drivers do not configure the output size of the sensors but the
> entire internal pipeline. Reflect this in the documentation.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Thanks
j
> ---
> .../userspace-api/media/drivers/camera-sensor.rst | 10 ++++++----
> 1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> index 9a1e8aa9fc77..bc55c861fb69 100644
> --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> @@ -10,11 +10,13 @@ used to control the camera sensor drivers.
>
> You may also find :ref:`media_writing_camera_sensor_drivers` useful.
>
> -Frame size
> -----------
> +Sensor internal pipeline configuration
> +--------------------------------------
>
> -There are two distinct ways to configure the frame size produced by camera
> -sensors.
> +Camera sensors have an internal processing pipeline including cropping and
> +binning functionality. The sensor drivers belong to two distinct classes, freely
> +configurable and register list-based drivers, depending on how the driver
> +configures this functionality.
>
> Freely configurable camera sensor drivers
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> --
> 2.39.5
>
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 6/9] media: uapi: Add V4L2_CID_CONFIG_MODEL control
2024-11-29 9:51 ` [RFC v3 6/9] media: uapi: Add V4L2_CID_CONFIG_MODEL control Sakari Ailus
@ 2024-12-05 8:43 ` Jacopo Mondi
2024-12-05 9:13 ` Sakari Ailus
0 siblings, 1 reply; 25+ messages in thread
From: Jacopo Mondi @ 2024-12-05 8:43 UTC (permalink / raw)
To: Sakari Ailus
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Sakari
On Fri, Nov 29, 2024 at 11:51:39AM +0200, Sakari Ailus wrote:
> Add the V4L2_CID_CONFIG_MODEL control for the configuration model.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
> .../userspace-api/media/v4l/ext-ctrls-image-process.rst | 4 ++++
> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 5 +++++
> include/uapi/linux/v4l2-controls.h | 3 +++
> 3 files changed, 12 insertions(+)
>
> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-image-process.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-image-process.rst
> index 27803dca8d3e..928e8e3eed7f 100644
> --- a/Documentation/userspace-api/media/v4l/ext-ctrls-image-process.rst
> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-image-process.rst
I was initially not sure this belongs to "Image process" controls, but
having had a look at the existing ones I'm not sure there are better
places.
I would have considered "Camera Controls" but I'm not sure it's -that-
better
> @@ -55,3 +55,7 @@ Image Process Control IDs
> control value divided by e.g. 0x100, meaning that to get no
> digital gain the control value needs to be 0x100. The no-gain
> configuration is also typically the default.
> +
> +``V4L2_CID_CONFIG_MODEL (bitmask)``
> + Which configuration models the sub-device supports. Please see
> + :ref:`media_subdev_config_model`.
Other RO controls in the file (such as PIXEL_RATE) specify "this
control is read-only"
> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> index 1ea52011247a..24c9c25e20d1 100644
> --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> @@ -1164,6 +1164,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> case V4L2_CID_TEST_PATTERN: return "Test Pattern";
> case V4L2_CID_DEINTERLACING_MODE: return "Deinterlacing Mode";
> case V4L2_CID_DIGITAL_GAIN: return "Digital Gain";
> + case V4L2_CID_CONFIG_MODEL: return "Sub-device configuration model";
>
> /* DV controls */
> /* Keep the order of the 'case's the same as in v4l2-controls.h! */
> @@ -1481,6 +1482,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
> case V4L2_CID_DV_RX_POWER_PRESENT:
> *type = V4L2_CTRL_TYPE_BITMASK;
> break;
> + case V4L2_CID_CONFIG_MODEL:
> + *flags |= V4L2_CTRL_FLAG_READ_ONLY;
> + *type = V4L2_CTRL_TYPE_BITMASK;
> + break;
> case V4L2_CID_MIN_BUFFERS_FOR_CAPTURE:
> case V4L2_CID_MIN_BUFFERS_FOR_OUTPUT:
> *type = V4L2_CTRL_TYPE_INTEGER;
> diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
> index 974fd254e573..0152240229ab 100644
> --- a/include/uapi/linux/v4l2-controls.h
> +++ b/include/uapi/linux/v4l2-controls.h
> @@ -1225,6 +1225,9 @@ enum v4l2_jpeg_chroma_subsampling {
> #define V4L2_CID_TEST_PATTERN (V4L2_CID_IMAGE_PROC_CLASS_BASE + 3)
> #define V4L2_CID_DEINTERLACING_MODE (V4L2_CID_IMAGE_PROC_CLASS_BASE + 4)
> #define V4L2_CID_DIGITAL_GAIN (V4L2_CID_IMAGE_PROC_CLASS_BASE + 5)
> +#define V4L2_CID_CONFIG_MODEL (V4L2_CID_IMAGE_PROC_CLASS_BASE + 6)
> +
> +#define V4L2_CID_CONFIG_MODEL_COMMON_RAW (1ULL << 0)
I wonder if "COMMON_" couldn't be removed.
Anyway, all nits or mostly questions, whatever is fine really
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Thanks
j
>
> /* DV-class control IDs defined by V4L2 */
> #define V4L2_CID_DV_CLASS_BASE (V4L2_CTRL_CLASS_DV | 0x900)
> --
> 2.39.5
>
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [RFC v3 6/9] media: uapi: Add V4L2_CID_CONFIG_MODEL control
2024-12-05 8:43 ` Jacopo Mondi
@ 2024-12-05 9:13 ` Sakari Ailus
0 siblings, 0 replies; 25+ messages in thread
From: Sakari Ailus @ 2024-12-05 9:13 UTC (permalink / raw)
To: Jacopo Mondi
Cc: linux-media, hverkuil, laurent.pinchart, Prabhakar, Kate Hsuan,
Alexander Shiyan, Mikhail Rudenko, Dave Stevenson,
Tommaso Merciai, Umang Jain, Benjamin Mugnier, Sylvain Petinot,
Christophe JAILLET, Julien Massot, Naushir Patuck, Yan, Dongcheng,
Cao, Bingbu, Qiu, Tian Shu, Wang, Hongju, Stefan Klug,
Mirela Rabulea, André Apitzsch, Heimir Thor Sverrisson,
Kieran Bingham, Stanislaw Gruszka, Mehdi Djait,
Ricardo Ribalda Delgado
Hi Jacopo,
On Thu, Dec 05, 2024 at 09:43:35AM +0100, Jacopo Mondi wrote:
> Hi Sakari
>
> On Fri, Nov 29, 2024 at 11:51:39AM +0200, Sakari Ailus wrote:
> > Add the V4L2_CID_CONFIG_MODEL control for the configuration model.
> >
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > ---
> > .../userspace-api/media/v4l/ext-ctrls-image-process.rst | 4 ++++
> > drivers/media/v4l2-core/v4l2-ctrls-defs.c | 5 +++++
> > include/uapi/linux/v4l2-controls.h | 3 +++
> > 3 files changed, 12 insertions(+)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-image-process.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-image-process.rst
> > index 27803dca8d3e..928e8e3eed7f 100644
> > --- a/Documentation/userspace-api/media/v4l/ext-ctrls-image-process.rst
> > +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-image-process.rst
>
> I was initially not sure this belongs to "Image process" controls, but
> having had a look at the existing ones I'm not sure there are better
> places.
Good question. Let's keep it here for the time being. I'm not sure any
other extended control class is a better target. Maybe old user controls? I
wonder what Hans would think.
>
> I would have considered "Camera Controls" but I'm not sure it's -that-
> better
>
> > @@ -55,3 +55,7 @@ Image Process Control IDs
> > control value divided by e.g. 0x100, meaning that to get no
> > digital gain the control value needs to be 0x100. The no-gain
> > configuration is also typically the default.
> > +
> > +``V4L2_CID_CONFIG_MODEL (bitmask)``
> > + Which configuration models the sub-device supports. Please see
> > + :ref:`media_subdev_config_model`.
>
> Other RO controls in the file (such as PIXEL_RATE) specify "this
> control is read-only"
I'll add that.
>
> > diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> > index 1ea52011247a..24c9c25e20d1 100644
> > --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> > +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> > @@ -1164,6 +1164,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> > case V4L2_CID_TEST_PATTERN: return "Test Pattern";
> > case V4L2_CID_DEINTERLACING_MODE: return "Deinterlacing Mode";
> > case V4L2_CID_DIGITAL_GAIN: return "Digital Gain";
> > + case V4L2_CID_CONFIG_MODEL: return "Sub-device configuration model";
> >
> > /* DV controls */
> > /* Keep the order of the 'case's the same as in v4l2-controls.h! */
> > @@ -1481,6 +1482,10 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
> > case V4L2_CID_DV_RX_POWER_PRESENT:
> > *type = V4L2_CTRL_TYPE_BITMASK;
> > break;
> > + case V4L2_CID_CONFIG_MODEL:
> > + *flags |= V4L2_CTRL_FLAG_READ_ONLY;
> > + *type = V4L2_CTRL_TYPE_BITMASK;
> > + break;
> > case V4L2_CID_MIN_BUFFERS_FOR_CAPTURE:
> > case V4L2_CID_MIN_BUFFERS_FOR_OUTPUT:
> > *type = V4L2_CTRL_TYPE_INTEGER;
> > diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
> > index 974fd254e573..0152240229ab 100644
> > --- a/include/uapi/linux/v4l2-controls.h
> > +++ b/include/uapi/linux/v4l2-controls.h
> > @@ -1225,6 +1225,9 @@ enum v4l2_jpeg_chroma_subsampling {
> > #define V4L2_CID_TEST_PATTERN (V4L2_CID_IMAGE_PROC_CLASS_BASE + 3)
> > #define V4L2_CID_DEINTERLACING_MODE (V4L2_CID_IMAGE_PROC_CLASS_BASE + 4)
> > #define V4L2_CID_DIGITAL_GAIN (V4L2_CID_IMAGE_PROC_CLASS_BASE + 5)
> > +#define V4L2_CID_CONFIG_MODEL (V4L2_CID_IMAGE_PROC_CLASS_BASE + 6)
> > +
> > +#define V4L2_CID_CONFIG_MODEL_COMMON_RAW (1ULL << 0)
>
> I wonder if "COMMON_" couldn't be removed.
We might have different models for raw sensors. These are just for the
current ones but new ones will have new functionality and we may need to
make changes that won't be compliant. Maybe in that case we'd have v2 or
something though.
>
> Anyway, all nits or mostly questions, whatever is fine really
> Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Thank you!
--
Kind regards,
Sakari Ailus
^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2024-12-05 9:13 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-11-29 9:51 [RFC v3 0/9] Sub-device configuration models Sakari Ailus
2024-11-29 9:51 ` [RFC v3 1/9] media: Documentation: Rework embedded data documentation Sakari Ailus
2024-11-29 9:51 ` [RFC v3 2/9] media: Documentation: Add a hyphen to list-based Sakari Ailus
2024-12-05 8:04 ` Jacopo Mondi
2024-11-29 9:51 ` [RFC v3 3/9] media: Documentation: Reword split of sensor driver to two classes Sakari Ailus
2024-12-05 8:06 ` Jacopo Mondi
2024-11-29 9:51 ` [RFC v3 4/9] media: Documentation: Add subdev configuration models, raw sensor model Sakari Ailus
2024-12-04 12:10 ` Jacopo Mondi
2024-12-04 13:22 ` Sakari Ailus
2024-12-04 13:37 ` Jacopo Mondi
2024-12-04 14:20 ` Sakari Ailus
2024-11-29 9:51 ` [RFC v3 5/9] media: Documentation: Add scaling and post-scaler crop for common raw Sakari Ailus
2024-12-04 11:25 ` Jacopo Mondi
2024-12-04 14:15 ` Sakari Ailus
2024-12-04 15:33 ` Jacopo Mondi
2024-12-05 6:56 ` Sakari Ailus
2024-11-29 9:51 ` [RFC v3 6/9] media: uapi: Add V4L2_CID_CONFIG_MODEL control Sakari Ailus
2024-12-05 8:43 ` Jacopo Mondi
2024-12-05 9:13 ` Sakari Ailus
2024-11-29 9:51 ` [RFC v3 7/9] media: uapi: Add V4L2_CID_BINNING control for binning configuration Sakari Ailus
2024-12-04 11:33 ` Jacopo Mondi
2024-12-04 14:24 ` Sakari Ailus
2024-11-29 9:51 ` [RFC v3 8/9] media: uapi: Add controls for sub-sampling configuration Sakari Ailus
2024-11-29 9:51 ` [RFC v3 9/9] media: Documentation: Add binning and sub-sampling controls Sakari Ailus
2024-12-04 11:40 ` [RFC v3 0/9] Sub-device configuration models Sakari Ailus
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox