Re: [RFC v5 04/15] media: Documentation: Add subdev configuration models, raw sensor model

Lad, Prabhakar
2025-02-03T15:14:38Z

Hi Sakari,

Thank you for the patch.

On Mon, Feb 3, 2025 at 8:59 AM Sakari Ailus
<sakari.ailus@linux.intel.com> wrote:
>
> Sub-device configuration models define what V4L2 API elements are
> available on a compliant sub-device and how do they behave.
>
> The patch also adds a model for common raw sensors.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
> .../media/drivers/camera-sensor.rst | 4 +
> .../media/v4l/common-raw-sensor.dia | 441 ++++++++++++++++++
> .../media/v4l/common-raw-sensor.svg | 134 ++++++
> .../userspace-api/media/v4l/dev-subdev.rst | 2 +
> .../media/v4l/subdev-config-model.rst | 209 +++++++++
> 5 files changed, 790 insertions(+)
> create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.dia
> create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.svg
> create mode 100644 Documentation/userspace-api/media/v4l/subdev-config-model.rst
>
> diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> index bc55c861fb69..5bc4c79d230c 100644
> --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> @@ -18,6 +18,8 @@ binning functionality. The sensor drivers belong to two distinct classes, freely
> configurable and register list-based drivers, depending on how the driver
> configures this functionality.
>
> +Also see :ref:`media_subdev_config_model_common_raw_sensor`.
> +
> Freely configurable camera sensor drivers
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> @@ -105,6 +107,8 @@ values programmed by the register sequences. The default values of these
> controls shall be 0 (disabled). Especially these controls shall not be inverted,
> independently of the sensor's mounting rotation.
>
> +.. _media_using_camera_sensor_drivers_embedded_data:
> +
> Embedded data
> -------------
<snip>
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index dcfcbd52490d..4d145bd3bd09 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -838,3 +838,5 @@ stream while it may be possible to enable and disable the embedded data stream.
>
> The embedded data format does not need to be configured on the sensor's pads as
> the format is dictated by the pixel data format in this case.
> +
> +.. include:: subdev-config-model.rst
> diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> new file mode 100644
> index 000000000000..b0bd09772ceb
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst
> @@ -0,0 +1,209 @@
> +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
> +
> +.. _media_subdev_config_model:
> +
> +Sub-device configuration models
> +===============================
> +
> +A sub-device configuration model specifies in detail what the user space can
> +expect from a sub-device in terms of V4L2 sub-device interface support,
> +semantics included.
> +
> +A sub-device may implement more than one configuration model at the same
> +time. The implemented configuration models can be obtained from the sub-device's
> +``V4L2_CID_CONFIG_MODEL`` control.
> +
> +.. _media_subdev_config_model_common_raw_sensor:
> +
> +Common raw camera sensor model
> +------------------------------
> +
> +The common raw camera sensor model defines a set of enumeration and
> +configuration interfaces (formats, selections etc.) that cover the vast majority
> +of functionality of raw camera sensors. Not all of the interfaces are
> +necessarily offered by all drivers.
> +
> +A sub-device complies with the common raw sensor model if the
> +``V4L2_CONFIG_MODEL_COMMON_RAW`` bit is set in the ``V4L2_CID_CONFIG_MODEL``
> +control of the sub-device.
> +
> +The common raw camera sensor model is aligned with
> +:ref:`media_using_camera_sensor_drivers`. Please refer to that regarding aspects
> +not specified here.
> +
> +Each camera sensor implementing the common raw sensor model exposes a single
> +V4L2 sub-device. The sub-device contains a single source pad (0) and two or more
> +internal pads: an image data internal pad (1) and optionally an embedded data
> +pad (2). Additionally, further internal pads may be supported for other
> +features, in which case they are documented separately for the given device.
> +
> +This is shown in :ref:`media_subdev_config_model_common_raw_sensor_subdev`.
> +
> +.. _media_subdev_config_model_common_raw_sensor_subdev:
> +
> +.. kernel-figure:: common-raw-sensor.svg
> + :alt: common-raw-sensor.svg
> + :align: center
> +
> + **Common raw sensor sub-device**
> +
> +Routes
> +^^^^^^
> +
> +A sub-device conforming to common raw camera sensor model implements the
> +following routes.
> +
> +.. flat-table:: Routes
> + :header-rows: 1
> +
> + * - Sink pad/stream
> + - Source pad/stream
> + - Static (X/M(aybe)/-)
> + - Mandatory (X/-)
> + - Synopsis
> + * - 1/0
> + - 0/0
> + - X
> + - X
> + - Image data
> + * - 2/0
> + - 0/1
> + - M
> + - \-
> + - Embedded data
> +
> +Support for the embedded data stream is optional. Drivers supporting the
> +embedded data stream may allow disabling and enabling the route when the
> +streaming is disabled.
> +
> +Sensor pixel array size, cropping and binning
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The sensor's pixel array is divided into one or more areas. The areas around the
> +edge of the pixel array, usually one or more sides, may contain optical black
> +pixels, dummy pixels and other non-image pixels. The entire pixel array size is
> +conveyed by the format on (pad, stream) pair 1/0.
> +
> +A rectangle within the pixel array contains the visible pixels. Capturing the
> +non-visible pixels outside the visible pixel area may be supported by the
> +sensor. The visible pixel area corresponds to the ``V4L2_SEL_TGT_CROP_DEFAULT``
> +selection target on (pad, stream) pair 1/0.
> +
> +Sensors can perform multiple operations that affect the output image size. First
> +of these is the analogue crop. Analogue crop limits the area of the pixel array
> +which the sensor will read, affecting sensor timing as well. The granularity of
> +the analogue crop configuration varies greatly across sensors: some sensors
> +support only a few different analogue crop configurations whereas others may
> +support anything divisible by a given number of pixels. The analogue crop
> +configuration corresponds the ``V4L2_SEL_TGT_CROP`` selection target on (pad,
s/ corresponds the/ corresponds to the

> +stream) pair 1/0. The default analogue crop rectangle corresponds to the visible
> +pixel area.
> +
> +In the next step, binning is performed on the image data read from camera
> +sensor's pixel array, as determined by the analogue crop configuration. Enabling
> +binning will effectively result in an image smaller than the original by given
> +binning factors horizontally and vertically. Typical values are 1/2 and 1/3 but
> +others may well be supported by the hardware as well.
> +
> +Sub-sampling follows binning. Sub-sampling, like binning, reduces the size of
> +the image by including only a subset of samples read from the sensor's pixel
> +matrix, typically every n'th pixel horizontally and vertically, taking the
> +sensor's colour pattern into account. Sub-sampling is generally configurable
> +separately horizontally and vertically.
> +
> +Binning and sub-sampling are configured using the ``V4L2_SEL_TGT_COMPOSE``
> +rectangle, relative to the analogue crop rectangle, on (pad, stream) pair
> +1/0. The driver implementation determines how to configure binning and
> +sub-sampling to achieve the desired size.
> +
> +The digital crop operation takes place after binning and sub-sampling. It is
> +configured by setting the ``V4L2_SEL_TGT_CROP`` rectangle on (pad, stream) pair
> +0/0. The resulting image size is further output by the sensor.
> +
> +The sensor's output mbus code is configured by setting the format on the (pad,
> +stream) pair 0/0. When setting the format, always use the same width and height
> +as for the digital crop setting.
> +
> +Drivers may only support some of even none of these configurations, in which
s/some of/some or

> +case they do not expose the corresponding selection rectangles. If any selection
> +targets are omitted, the further selection rectangle or format is instead
> +related to the previous implemented selection rectangle. For instance, if the
> +sensor supports binning but not analogue crop, then the binning configuration
> +(``V4L2_SEL_TGT_COMPOSE`` selection target) is done in relation to the visible
> +pixel area (``V4L2_SEL_TGT_CROP_DEFAULT`` selection target).
> +
> +Also refer to :ref:`Selection targets <v4l2-selection-targets-table>`.
> +
> +.. flat-table:: Selection targets on pads
> + :header-rows: 1
> +
> + * - Pad/Stream
> + - Selection target/format
> + - Mandatory (X/-)
> + - Modifiable (X/-)
> + - Synopsis
> + * - 1/0
> + - Format
> + - X
> + - \-
> + - Image data format. The width and the height fields indicates the full
> + size of the pixel array, including non-visible pixels. The media bus
> + code of this format reflects the native pixel depth of the sensor.
> + * - 1/0
> + - ``V4L2_SEL_TGT_CROP_DEFAULT``
> + - X
> + - \
> + - The visible pixel area. This rectangle is relative to the format on the
> + same (pad, stream).
> + * - 1/0
> + - ``V4L2_SEL_TGT_CROP``
> + - \-
> + - X
> + - Analogue crop. Analogue crop typically has a coarse granularity. This
> + rectangle is relative to the format on the same (pad, stream).
> + * - 1/0
> + - ``V4L2_SEL_TGT_COMPOSE``
> + - \-
> + - X
> + - Binning and sub-sampling. This rectangle is relative to the
> + ``V4L2_SEL_TGT_CROP`` rectangle on the same (pad, stream). The
> + combination of binning and sub-sampling is configured using this
> + selection target.
> + * - 2/0
> + - Format
> + - X
> + - \-
> + - Embedded data format.
> + * - 0/0
> + - ``V4L2_SEL_TGT_CROP``
> + - \-
> + - X
> + - Digital crop. This rectangle is relative to the ``V4L2_SEL_TGT_COMPOSE``
> + rectangle on (pad, stream) pair 1/0.
> + * - 0/0
> + - Format
> + - X
> + - X
> + - Image data source format. Always assign the width and height fields of
> + the format to the same values than for the ``V4L2_SEL_TGT_CROP``
> + rectangle on (pad, stream) pair 0/0. The media bus code reflects the
> + pixel data output of the sensor.
> + * - 0/1
> + - Format
> + - X
> + - \-
> + - Embedded data source format.
> +
> +Embedded data
> +^^^^^^^^^^^^^
> +
> +The embedded data stream is produced by the sensor when the corresponding route
> +is enabled. The embedded data route may also be immutable or not exist at all,
> +in case the sensor (or the driver) does not support it.
> +
> +Generally the sensor embedded data width is determined by the width of the image
> +data whereas the number of lines are constant for the embedded data. The user
> +space may obtain the size of the embedded data once the image data size on the
> +source pad has been configured.
> +
> +Also see :ref:`media_using_camera_sensor_drivers_embedded_data`.
> --
> 2.39.5
>
Reviewed-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>

Cheers,
Prabhakar