From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=192.198.163.17 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1795008543; b=cR23TD2ZEKBAhMFlQC2mZQ2N94MH+85GNCY4vQUndCJ49FkS9MGMK8lyBYh2FKlVRpVe BMLTkH1NuUTZO2ZqmtIxU3lTJE5Cl6OBH3iS6AxJ2pWC9eM9DLECqveS7ykPEs9Xs528O oSTTSeml3W1Bci+zVIoVUIx9a6Yk9fiG99/bw0XDJ5zPbq/t+GbtDSCaRvjeRNOc8uUof aT/AhUiTgaLsixQ9pu6Yz3h5Snyi8H1c51toj3IhYVlg4bXWUBpWQHR6nrFA+ovpdNLOg Vn2R00Z00EYwtqky4T61df892p7WgVhO2iCywcyY5/AGvGSOLoaeIIyj7OVdUwrOmVQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1795008543; h=DMARC-Filter:DKIM-Signature:X-CSE-ConnectionGUID:X-CSE-MsgGUID: X-IronPort-AV:X-IronPort-AV:Received:X-CSE-ConnectionGUID: X-CSE-MsgGUID:X-ExtLoop1:X-IronPort-AV:Received:Received:Received: Organization:From:To:Cc:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:MIME-Version:Content-Transfer-Encoding; bh=0CgtiD7GhqVv1gjphyq6jIZ3MEMdKOAikkj+QmkqbdE=; b=sB0Q9BdNCPrcJC4HItUaS+t0ZtH8jLt+ZhJxuHEf2UrAbH+2mOFYHUo7ur2HH/EOtIhR 8eDhdB+oVnWlfj8s2C/jUB+Zf6a5vMhJkKD8yzmMzyAbsHHx2ryfg8yPYfCCfuhMz98cp cl0975kX+uCqe41CfWOumOqQ3pj6XI/jgfhSeN6VWox6NPDAmV8D6uo6zo8aS0Ti4/eRK oIcBDqyqjshdpxf0whd/dbkURrSZb4LMc41Dmr4OlSDRn3qGhstgyyfGJ5fObPfrhG0zV Yb992eHcM5tncvwtjR6NflXgCChrSwle33vIo/V6wJcSyuPWnxRWpRQxmUmN8TLViWw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=neA4kr+U; arc=none smtp.remote-ip=192.198.163.17 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA5FE60A2616 for ; Wed, 18 Nov 2026 13:29:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org BA5FE60A2616 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=neA4kr+U DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1795008542; x=1826544542; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ISe0sTcKfm6D9fHIJ1TQ8JeyBp/YwptkQQWRXbu6d1c=; b=neA4kr+UYNoDlijfvPWmZUP2B96E17XF+VCubWEgOflUm2za32HvToGo B0R+StbS+DcDgHYFMrq3cF016nIpLrvZy66MW2h/j5qPjWyQbIN1Ga0Cr jzcB9w7DXUHXmqHz6kkkx9oSZigmnZjdkeFbMdOj5WLOGNhisseGqcKOG KZCn7usMc3WVXccdYxxNepBmXSTOYRq+Ye7kxnwcyoQeA1ZgIdHbj42sc ZCUWUyCRMpOAOFPE3Z2gIEEvA1Fpk6+mfJOt02fnlFupSDLqrGz0eE5+i a89pDxKRif6DJe5pgqed4IruO5f0T3mO1VzFbjWnj/4UopRI/yWQRqmNV g==; X-CSE-ConnectionGUID: rIHBf2YpRc6IE2arwsFvTA== X-CSE-MsgGUID: PEguUZNVSlGpN1nue/DN7w== X-IronPort-AV: E=McAfee;i="6800,10657,11969"; a="3808798" X-IronPort-AV: E=Sophos;i="6.27,219,1787036400"; d="scan'208";a="3808798" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Nov 2026 05:28:57 -0800 X-CSE-ConnectionGUID: qCOhwB1qTC+Nujsq2BWDeQ== X-CSE-MsgGUID: 7P9k+UzhQC+j3FqvraZzyw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.27,219,1787036400"; d="scan'208";a="10376719" Received: from ettammin-mobl3.ger.corp.intel.com (HELO kekkonen.fi.intel.com) ([10.245.244.183]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Nov 2026 05:28:49 -0800 Received: from punajuuri.localdomain (unknown [192.168.240.130]) by kekkonen.fi.intel.com (Postfix) with ESMTP id 67DD4122F58; Wed, 18 Nov 2026 15:28:31 +0200 (EET) Received: from sailus by punajuuri.localdomain with local (Exim 4.98.2) (envelope-from ) id 1xTfi0-00000000tb7-08rd; Wed, 18 Nov 2026 15:28:20 +0200 Organization: Intel Finland Oy - BIC 0357606-4 - c/o Alberga Business Park, 6 krs, Bertel Jungin Aukio 5, 02600 Espoo From: Sakari Ailus To: linux-media@vger.kernel.org Cc: hans@jjverkuil.nl, laurent.pinchart@ideasonboard.com, Prabhakar , Kate Hsuan , Dave Stevenson , Tommaso Merciai , Benjamin Mugnier , Sylvain Petinot , Christophe JAILLET , Julien Massot , Naushir Patuck , "Yan, Dongcheng" , Stefan Klug , Mirela Rabulea , =?UTF-8?q?Andr=C3=A9=20Apitzsch?= , Heimir Thor Sverrisson , Kieran Bingham , Mehdi Djait , Ricardo Ribalda Delgado , Hans de Goede , Jacopo Mondi , Tomi Valkeinen , David Plowman , "Yu, Ong Hock" , "Ng, Khai Wen" , Jai Luthra , Rishikesh Donadkar , Mattijs Korpershoek , Antti Laakso , "Bajpai, Manik" , Divyamani Tripathi Subject: [PATCH v13 23/73] media: Documentation: Add subdev configuration models, raw sensor model Date: Wed, 18 Nov 2026 15:27:29 +0200 Message-ID: <20261118132819.213473-24-sakari.ailus@linux.intel.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261118132819.213473-1-sakari.ailus@linux.intel.com> References: <20261118132819.213473-1-sakari.ailus@linux.intel.com> X-Mailing-List: linux-media@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sub-device configuration models define what V4L2 API elements are available on a compliant sub-device and how do they behave. The patch also adds a model for common raw sensors. Signed-off-by: Sakari Ailus Reviewed-by: Tomi Valkeinen Reviewed-by: Lad Prabhakar Reviewed-by: Mirela Rabulea Reviewed-by: Jacopo Mondi --- .../media/drivers/camera-sensor.rst | 4 + .../media/v4l/common-raw-sensor.dia | 442 ++++++++++++++++++ .../media/v4l/common-raw-sensor.svg | 134 ++++++ .../userspace-api/media/v4l/dev-subdev.rst | 9 +- .../userspace-api/media/v4l/devices.rst | 1 + .../media/v4l/subdev-config-model.rst | 250 ++++++++++ 6 files changed, 836 insertions(+), 4 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor= .dia create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor= .svg create mode 100644 Documentation/userspace-api/media/v4l/subdev-config-mod= el.rst diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/= Documentation/userspace-api/media/drivers/camera-sensor.rst index 87f86a2f6cc5..8002893f1c36 100644 --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst @@ -18,6 +18,8 @@ binning functionality. The sensor drivers belong to two d= istinct classes, freely configurable and register list-based drivers, depending on how the driver configures this functionality. =20 +Also see :ref:`media_subdev_config_model_common_raw_sensor`. + Freely configurable camera sensor drivers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 @@ -118,6 +120,8 @@ values programmed by the register sequences. The defaul= t values of these controls shall be 0 (disabled). Especially these controls shall not be inv= erted, independently of the sensor's mounting rotation. =20 +.. _media_using_camera_sensor_drivers_embedded_data: + Embedded data ------------- =20 diff --git a/Documentation/userspace-api/media/v4l/common-raw-sensor.dia b/= Documentation/userspace-api/media/v4l/common-raw-sensor.dia new file mode 100644 index 000000000000..24b3f2b2a626 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/common-raw-sensor.dia @@ -0,0 +1,442 @@ + + + + + + + + + + + + + #A4# + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + #image data (1)# + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + #embedded data (n - 1)# + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + #source pad (0)# + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Documentation/userspace-api/media/v4l/common-raw-sensor.svg b/= Documentation/userspace-api/media/v4l/common-raw-sensor.svg new file mode 100644 index 000000000000..1d6055da2519 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/common-raw-sensor.svg @@ -0,0 +1,134 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documen= tation/userspace-api/media/v4l/dev-subdev.rst index 9168a3a45c9e..3472b0591d79 100644 --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst @@ -721,7 +721,8 @@ To configure this pipeline, the userspace must take the= following steps: :ref:`VIDIOC_SUBDEV_S_FMT ` ioctls to configure ea= ch stream endpoint in each sub-device. =20 - In case generic raw and metadata formats are used, :ref:`V4L2_CID_CFA_P= ATTERN - ` and :ref:`V4L2_CID_METADATA_LAYOUT - ` controls are present on the sou= rce - sub-device to obtain the pixel array CFA pattern and metadata layout. + In case generic raw and metadata formats are used, + :ref:`V4L2_CID_CFA_PATTERN ` and + :ref:`V4L2_CID_METADATA_LAYOUT ` + controls are present on the source sub-device to obtain the pixel array= CFA + pattern and metadata layout. diff --git a/Documentation/userspace-api/media/v4l/devices.rst b/Documentat= ion/userspace-api/media/v4l/devices.rst index d4790b9ef81a..4d6b7b55b0b3 100644 --- a/Documentation/userspace-api/media/v4l/devices.rst +++ b/Documentation/userspace-api/media/v4l/devices.rst @@ -24,4 +24,5 @@ Interfaces dev-touch dev-event dev-subdev + subdev-config-model dev-meta diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst = b/Documentation/userspace-api/media/v4l/subdev-config-model.rst new file mode 100644 index 000000000000..cbd12986b977 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst @@ -0,0 +1,250 @@ +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later + +.. _media_subdev_config_model: + +Sub-device configuration models +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D + +The V4L2 specification defines a subdev API that exposes three type of +configuration elements: formats, selection rectangles and controls. The +specification contains generic information about how those configuration +elements behave, but not precisely how they apply to particular hardware +features. We leave some leeway to drivers to decide how to map selection +rectangles to device features, as long as they comply with the V4L2 +specification. This is needed as hardware features differ between devices,= so +it's the driver's responsibility to handle this mapping. + +Unfortunately, this lack of clearly defined mapping in the specification h= as led +to different drivers mapping the same hardware features to different API +elements, or implementing the API elements with slightly different +behaviours. Furthermore, many drivers have implemented selection rectangle= s in +ways that do not comply with the V4L2 specification. All of this makes use= rspace +development difficult. + +Sub-device configuration models specify in detail what the user space can = expect +from a sub-device in terms of V4L2 sub-device interface support, semantics +included. + +A sub-device may implement more than one configuration model at the same +time. The implemented configuration models can be obtained from the sub-de= vice's +``V4L2_CID_CONFIG_MODEL`` control. + +.. _media_subdev_config_model_common_raw_sensor: + +Common raw camera sensor model +------------------------------ + +The common raw camera sensor model defines a set of enumeration and +configuration interfaces (formats, selections etc.) that cover the vast ma= jority +of functionality of raw camera sensors. Not all of the interfaces are +necessarily offered by all drivers. + +A sub-device complies with the common raw sensor model if the +``V4L2_CONFIG_MODEL_COMMON_RAW_SENSOR`` bit is set in the +``V4L2_CID_CONFIG_MODEL`` control of the sub-device. + +The common raw camera sensor model is aligned with +:ref:`media_using_camera_sensor_drivers`. Please refer to that regarding a= spects +not specified here. + +Each camera sensor implementing the common raw sensor model exposes a sing= le +V4L2 sub-device. The sub-device contains a single source pad (0) and two o= r more +internal pads: one or more image data internal pads (starting from 1) and +optionally an embedded data pad. + +Additionally, further internal pads may be supported for other features. U= sing +more than one image data internal pad or more than one non-image data pad +requires these pads documented separately for the given device. The indice= s of +the image data internal pads shall be lower than those of the non-image da= ta +pads. + +This is shown in :ref:`media_subdev_config_model_common_raw_sensor_subdev`. + +.. _media_subdev_config_model_common_raw_sensor_subdev: + +.. kernel-figure:: common-raw-sensor.svg + :alt: common-raw-sensor.svg + :align: center + + **Common raw sensor sub-device with two pads** + +Routes +^^^^^^ + +A sub-device conforming to common raw camera sensor model implements the +following routes. + +.. flat-table:: Routes + :header-rows: 1 + + * - Sink pad/stream + - Source pad/stream + - Static (X/M(aybe)/-) + - Mandatory (X/-) + - Synopsis + * - 1/0 + - 0/0 + - X + - X + - Image data + * - 2/0 + - 0/1 + - M + - \- + - Embedded data + +Support for the embedded data stream is optional. Drivers supporting the +embedded data stream may allow disabling and enabling the route. + +Sensor pixel array size, cropping and binning +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The sensor's pixel array is divided into one or more areas. The areas arou= nd the +the visible pixel area in the pixel array, usually one or more sides, may +contain optical black pixels, dummy pixels and other non-image pixels. The= full +size of the pixel array, including pixels that cannot be captured, is conv= eyed +by the format on (pad, stream) pair 1/0. + +Capturing the non-visible pixels outside the visible pixel area may be sup= ported +by the sensor. This area corresponds to the ``V4L2_SEL_TGT_CROP_BOUNDS`` +selection target on (pad, stream) pair 1/0. Non-visible pixels may include +optical black pixels, dummy pixels or other non-visible pixels. The visible +pixel area corresponds to the ``V4L2_SEL_TGT_CROP_DEFAULT`` selection targ= et on +(pad, stream) pair 1/0. + +A rectangle within the pixel array contains the visible pixels. Capturing = the +non-visible pixels outside the visible pixel area may be supported by the +sensor. The visible pixel area corresponds to the ``V4L2_SEL_TGT_CROP_DEFA= ULT`` +selection target on (pad, stream) pair 1/0. + +Sensors can perform multiple operations that affect the output image size.= The first +of these is the analogue crop. Analogue crop limits the area of the pixel = array +which the sensor will read, affecting sensor timing as well. The granulari= ty of +the analogue crop configuration varies greatly across sensors: some sensors +support only a few different analogue crop configurations whereas others m= ay +support anything divisible by a given number of pixels. The analogue crop +configuration corresponds to the ``V4L2_SEL_TGT_CROP`` selection target on= (pad, +stream) pair 1/0. The default analogue crop rectangle corresponds to the v= isible +pixel area. + +In the next step, binning is performed on the image data read from camera +sensor's pixel array, as determined by the analogue crop configuration. En= abling +binning will effectively result in an image smaller than the original by g= iven +binning factors horizontally and vertically. Typical values are 1/2 and 1/= 3 but +others may well be supported by the hardware as well. + +Sub-sampling follows binning. Sub-sampling, like binning, reduces the size= of +the image by including only a subset of samples read from the sensor's pix= el +matrix, typically every n'th pixel horizontally and vertically, taking the +sensor's color pattern into account. Sub-sampling is generally configurable +separately horizontally and vertically. + +The combined effect of binning and sub-sampling is configured using the +``V4L2_SEL_TGT_COMPOSE`` rectangle, relative to the analogue crop rectangl= e, on +(pad, stream) pair 1/0. The driver implementation determines how to config= ure +binning and sub-sampling to achieve the desired size. + +The digital crop operation takes place after binning and sub-sampling. It = is +configured by setting the ``V4L2_SEL_TGT_CROP`` rectangle on (pad, stream)= pair +0/0. The resulting image size is further output by the sensor on the senso= r's +data interface. + +The sensor's output mbus code is configured by setting the format on the (= pad, +stream) pair 0/0. When setting the format, always use the same width and h= eight +as for the digital crop setting. + +Drivers may only support some or even none of these configurations, in whi= ch +case they do not expose the corresponding selection rectangles. If any sel= ection +targets are omitted, the further selection rectangle or format is instead +related to the previous implemented selection rectangle. For instance, if = the +sensor supports binning but not analogue crop, then the binning configurat= ion +(``V4L2_SEL_TGT_COMPOSE`` selection target) is done in relation to the vis= ible +pixel area (``V4L2_SEL_TGT_CROP_DEFAULT`` selection target). + +Also refer to :ref:`Selection targets `. + +.. flat-table:: Selection targets on pads + :header-rows: 1 + + * - Pad/Stream + - Selection target/format + - Mandatory (X/-) + - Modifiable (X/-) + - Synopsis + * - 1/0 + - Format + - X + - \- + - The width and the height fields indicates the full size of the phy= sical + pixel array, including non-visible pixels and pixels that cannot be + captured. The media bus code of this format reflects the native pi= xel + depth of the sensor. + * - 1/0 + - ``V4L2_SEL_TGT_CROP_BOUNDS`` + - X + - \- + - The width and the height fields indicates the full size of the pix= el + array that can be captured, including non-visible pixels. Should t= he + shape of the pixel array not be a rectangle, this rectangle is the + smallest possible rectangle that covers all capturable pixels of t= he + pixel array. This rectangle is relative to the format on the same = (pad, + stream). + * - 1/0 + - ``V4L2_SEL_TGT_CROP_DEFAULT`` + - X + - \- + - The visible pixel area. This rectangle is relative to the format o= n the + same (pad, stream). + * - 1/0 + - ``V4L2_SEL_TGT_CROP`` + - \- + - X + - Analogue crop. Analogue crop typically has a coarse granularity. T= his + rectangle is relative to the format on the same (pad, stream). + * - 1/0 + - ``V4L2_SEL_TGT_COMPOSE`` + - \- + - \- + - Binning and sub-sampling. This rectangle is relative to the + ``V4L2_SEL_TGT_CROP`` rectangle on the same (pad, stream). The + combination of binning and sub-sampling is expressed with this sel= ection + target. + * - 2/0 + - Format + - X + - \- + - Embedded data format. + * - 0/0 + - ``V4L2_SEL_TGT_CROP`` + - \- + - X + - Digital crop. This rectangle is relative to the ``V4L2_SEL_TGT_COM= POSE`` + rectangle on (pad, stream) pair 1/0. + * - 0/0 + - Format + - X + - X + - Image data source format. Always assign the width and height field= s of + the format to the same values than for the ``V4L2_SEL_TGT_CROP`` + rectangle on (pad, stream) pair 0/0. The media bus code reflects t= he + pixel data output of the sensor. Setting the media bus code on thi= s pad + configures the output format. + * - 0/1 + - Format + - X + - \- + - Embedded data source format. + +Embedded data +^^^^^^^^^^^^^ + +The embedded data stream is produced by the sensor when the corresponding = route +is enabled. The embedded data route may also be immutable or not exist at = all, +in case the sensor (or the driver) does not support it. + +Generally the sensor embedded data width is determined by the width of the= image +data whereas the number of lines are constant for the embedded data. The u= ser +space may obtain the size of the embedded data once the image data size on= the +source pad has been configured. + +Also see :ref:`media_using_camera_sensor_drivers_embedded_data`. --=20 2.47.3