From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D84542061; Fri, 8 Mar 2024 10:41:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=213.167.242.64 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709894518; cv=none; b=MbE7CcvhgEmvaq6uTg9nTwFN/9k5YnGou0BU6paPpDgCIYYi8TIfbIm8qNdW9Ae9QqYefI1u1ORQOiYa3l09S22rdi6Tz687p2YeImStTpPSUBvHwJQeHsabIjt2bvXmVaKlTw7g156IkRoF/WmXbopI6/03hzuwNQYnGqyYCtc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709894518; c=relaxed/simple; bh=lGCU6S2X6aX2lvBwOqxetR9pkjOAwF78pZdbdutbuQ8=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=tEHczLYoeCjzuodYWTwz6N+BJbFwRljUyrNPV5dyQw3gKJcObu9c+aF6BhT/wpV1LXkzm60KxuAx+4i3Aco+SfUbutD8rhs66gB40ioOTs/MPDWvnRLFKfVIsxAmouOwtJdJOFHzk76gDYQZGS1mXFMu/NNASEeGVPMsA0ZQ9IY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ideasonboard.com; spf=pass smtp.mailfrom=ideasonboard.com; dkim=pass (1024-bit key) header.d=ideasonboard.com header.i=@ideasonboard.com header.b=jvAD49JY; arc=none smtp.client-ip=213.167.242.64 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ideasonboard.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ideasonboard.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="jvAD49JY" Received: from [192.168.0.43] (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 1EBD1552; Fri, 8 Mar 2024 11:41:35 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1709894495; bh=lGCU6S2X6aX2lvBwOqxetR9pkjOAwF78pZdbdutbuQ8=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=jvAD49JYrA9fwcVnxA1peQDqOOPhIju6eIGD1uHx2V6t0n88K468Dy1bqJwfdl+T4 wTZ8uY4OhlIrc4eox0DPxGD2zga6muB3ezWV1vZWQr16VwJY9cT7Nn7YHcvxA07xD2 NvBvWK0Ho56vQDtPsYOLsLpeFYuTnn9zOQky6De0= Message-ID: <496b70b3-0c0b-4957-8891-4151a2e5b501@ideasonboard.com> Date: Fri, 8 Mar 2024 10:41:51 +0000 Precedence: bulk X-Mailing-List: linux-media@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 4/5] media: Documentation: Add Mali-C55 ISP Documentation Content-Language: en-US To: Jacopo Mondi Cc: linux-media@vger.kernel.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, nayden.kanchev@arm.com, robh+dt@kernel.org, mchehab@kernel.org, krzysztof.kozlowski+dt@linaro.org, conor+dt@kernel.org, jerome.forissier@linaro.org, kieran.bingham@ideasonboard.com, laurent.pinchart@ideasonboard.com References: <20240305164832.2055437-1-dan.scally@ideasonboard.com> <20240305164832.2055437-5-dan.scally@ideasonboard.com> <545lcuehnm2yvzdjpcktsyo45n42thgfhqbegrelhc66mdkumj@utpfj5gzkcnm> From: Dan Scally Autocrypt: addr=dan.scally@ideasonboard.com; keydata= xsFNBGLydlEBEADa5O2s0AbUguprfvXOQun/0a8y2Vk6BqkQALgeD6KnXSWwaoCULp18etYW B31bfgrdphXQ5kUQibB0ADK8DERB4wrzrUb5CMxLBFE7mQty+v5NsP0OFNK9XTaAOcmD+Ove eIjYvqurAaro91jrRVrS1gBRxIFqyPgNvwwL+alMZhn3/2jU2uvBmuRrgnc/e9cHKiuT3Dtq MHGPKL2m+plk+7tjMoQFfexoQ1JKugHAjxAhJfrkXh6uS6rc01bYCyo7ybzg53m1HLFJdNGX sUKR+dQpBs3SY4s66tc1sREJqdYyTsSZf80HjIeJjU/hRunRo4NjRIJwhvnK1GyjOvvuCKVU RWpY8dNjNu5OeAfdrlvFJOxIE9M8JuYCQTMULqd1NuzbpFMjc9524U3Cngs589T7qUMPb1H1 NTA81LmtJ6Y+IV5/kiTUANflpzBwhu18Ok7kGyCq2a2jsOcVmk8gZNs04gyjuj8JziYwwLbf vzABwpFVcS8aR+nHIZV1HtOzyw8CsL8OySc3K9y+Y0NRpziMRvutrppzgyMb9V+N31mK9Mxl 1YkgaTl4ciNWpdfUe0yxH03OCuHi3922qhPLF4XX5LN+NaVw5Xz2o3eeWklXdouxwV7QlN33 u4+u2FWzKxDqO6WLQGjxPE0mVB4Gh5Pa1Vb0ct9Ctg0qElvtGQARAQABzShEYW4gU2NhbGx5 IDxkYW4uc2NhbGx5QGlkZWFzb25ib2FyZC5jb20+wsGNBBMBCAA3FiEEsdtt8OWP7+8SNfQe kiQuh/L+GMQFAmLydlIFCQWjmoACGwMECwkIBwUVCAkKCwUWAgMBAAAKCRCSJC6H8v4YxDI2 EAC2Gz0iyaXJkPInyshrREEWbo0CA6v5KKf3I/HlMPqkZ48bmGoYm4mEQGFWZJAT3K4ir8bg cEfs9V54gpbrZvdwS4abXbUK4WjKwEs8HK3XJv1WXUN2bsz5oEJWZUImh9gD3naiLLI9QMMm w/aZkT+NbN5/2KvChRWhdcha7+2Te4foOY66nIM+pw2FZM6zIkInLLUik2zXOhaZtqdeJZQi HSPU9xu7TRYN4cvdZAnSpG7gQqmLm5/uGZN1/sB3kHTustQtSXKMaIcD/DMNI3JN/t+RJVS7 c0Jh/ThzTmhHyhxx3DRnDIy7kwMI4CFvmhkVC2uNs9kWsj1DuX5kt8513mvfw2OcX9UnNKmZ nhNCuF6DxVrL8wjOPuIpiEj3V+K7DFF1Cxw1/yrLs8dYdYh8T8vCY2CHBMsqpESROnTazboh AiQ2xMN1cyXtX11Qwqm5U3sykpLbx2BcmUUUEAKNsM//Zn81QXKG8vOx0ZdMfnzsCaCzt8f6 9dcDBBI3tJ0BI9ByiocqUoL6759LM8qm18x3FYlxvuOs4wSGPfRVaA4yh0pgI+ModVC2Pu3y ejE/IxeatGqJHh6Y+iJzskdi27uFkRixl7YJZvPJAbEn7kzSi98u/5ReEA8Qhc8KO/B7wprj xjNMZNYd0Eth8+WkixHYj752NT5qshKJXcyUU87BTQRi8nZSARAAx0BJayh1Fhwbf4zoY56x xHEpT6DwdTAYAetd3yiKClLVJadYxOpuqyWa1bdfQWPb+h4MeXbWw/53PBgn7gI2EA7ebIRC PJJhAIkeym7hHZoxqDQTGDJjxFEL11qF+U3rhWiL2Zt0Pl+zFq0eWYYVNiXjsIS4FI2+4m16 tPbDWZFJnSZ828VGtRDQdhXfx3zyVX21lVx1bX4/OZvIET7sVUufkE4hrbqrrufre7wsjD1t 8MQKSapVrr1RltpzPpScdoxknOSBRwOvpp57pJJe5A0L7+WxJ+vQoQXj0j+5tmIWOAV1qBQp hyoyUk9JpPfntk2EKnZHWaApFp5TcL6c5LhUvV7F6XwOjGPuGlZQCWXee9dr7zym8iR3irWT +49bIh5PMlqSLXJDYbuyFQHFxoiNdVvvf7etvGfqFYVMPVjipqfEQ38ST2nkzx+KBICz7uwj JwLBdTXzGFKHQNckGMl7F5QdO/35An/QcxBnHVMXqaSd12tkJmoRVWduwuuoFfkTY5mUV3uX xGj3iVCK4V+ezOYA7c2YolfRCNMTza6vcK/P4tDjjsyBBZrCCzhBvd4VVsnnlZhVaIxoky4K aL+AP+zcQrUZmXmgZjXOLryGnsaeoVrIFyrU6ly90s1y3KLoPsDaTBMtnOdwxPmo1xisH8oL a/VRgpFBfojLPxMAEQEAAcLBfAQYAQgAJhYhBLHbbfDlj+/vEjX0HpIkLofy/hjEBQJi8nZT BQkFo5qAAhsMAAoJEJIkLofy/hjEXPcQAMIPNqiWiz/HKu9W4QIf1OMUpKn3YkVIj3p3gvfM Res4fGX94Ji599uLNrPoxKyaytC4R6BTxVriTJjWK8mbo9jZIRM4vkwkZZ2bu98EweSucxbp vjESsvMXGgxniqV/RQ/3T7LABYRoIUutARYq58p5HwSP0frF0fdFHYdTa2g7MYZl1ur2JzOC FHRpGadlNzKDE3fEdoMobxHB3Lm6FDml5GyBAA8+dQYVI0oDwJ3gpZPZ0J5Vx9RbqXe8RDuR du90hvCJkq7/tzSQ0GeD3BwXb9/R/A4dVXhaDd91Q1qQXidI+2jwhx8iqiYxbT+DoAUkQRQy xBtoCM1CxH7u45URUgD//fxYr3D4B1SlonA6vdaEdHZOGwECnDpTxecENMbz/Bx7qfrmd901 D+N9SjIwrbVhhSyUXYnSUb8F+9g2RDY42Sk7GcYxIeON4VzKqWM7hpkXZ47pkK0YodO+dRKM yMcoUWrTK0Uz6UzUGKoJVbxmSW/EJLEGoI5p3NWxWtScEVv8mO49gqQdrRIOheZycDmHnItt 9Qjv00uFhEwv2YfiyGk6iGF2W40s2pH2t6oeuGgmiZ7g6d0MEK8Ql/4zPItvr1c1rpwpXUC1 u1kQWgtnNjFHX3KiYdqjcZeRBiry1X0zY+4Y24wUU0KsEewJwjhmCKAsju1RpdlPg2kC In-Reply-To: <545lcuehnm2yvzdjpcktsyo45n42thgfhqbegrelhc66mdkumj@utpfj5gzkcnm> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi Jacopo On 08/03/2024 10:23, Jacopo Mondi wrote: > Hi Dan > > On Wed, Mar 06, 2024 at 10:03:01AM +0000, Dan Scally wrote: >> Hi Jacopo - thanks for the review >> >> On 05/03/2024 17:24, Jacopo Mondi wrote: >>> Hi Dan >>> >>> On Tue, Mar 05, 2024 at 04:48:31PM +0000, Daniel Scally wrote: >>>> Add a documentation page for the mali-c55 driver, which gives a brief >>>> overview of the hardware and explains how to use the driver's capture >>>> devices and the crop/scaler functions. >>>> >>>> Acked-by: Nayden Kanchev >>>> Signed-off-by: Daniel Scally >>>> --- >>>> Changes in v3: >>>> - Documented the synchronised buffer sequence numbers (Sakari) >>>> - Clarified that the downscale pipe cannot output raw data, the ISP'S >>>> resolution limits and choice of media bus format code (Kieran) >>>> >>>> Changes in v2: >>>> >>>> - none >>>> >>>> .../admin-guide/media/mali-c55-graph.dot | 19 + >>>> Documentation/admin-guide/media/mali-c55.rst | 330 ++++++++++++++++++ >>>> .../admin-guide/media/v4l-drivers.rst | 1 + >>>> 3 files changed, 350 insertions(+) >>>> create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot >>>> create mode 100644 Documentation/admin-guide/media/mali-c55.rst >>>> >>>> diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot b/Documentation/admin-guide/media/mali-c55-graph.dot >>>> new file mode 100644 >>>> index 000000000000..0775ba42bf4c >>>> --- /dev/null >>>> +++ b/Documentation/admin-guide/media/mali-c55-graph.dot >>>> @@ -0,0 +1,19 @@ >>>> +digraph board { >>>> + rankdir=TB >>>> + n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | { 0}}", shape=Mrecord, style=filled, fillcolor=green] >>>> + n00000001:port0 -> n00000003:port0 [style=dashed] >>>> + n00000003 [label="{{ 0} | mali-c55 isp\n/dev/v4l-subdev1 | { 1 | 2}}", shape=Mrecord, style=filled, fillcolor=green] >>>> + n00000003:port1 -> n00000007:port0 [style=bold] >>>> + n00000003:port2 -> n00000007:port2 [style=bold] >>>> + n00000003:port1 -> n0000000b:port0 [style=bold] >>>> + n00000007 [label="{{ 0 | 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | { 1}}", shape=Mrecord, style=filled, fillcolor=green] >>>> + n00000007:port1 -> n0000000e [style=bold] >>>> + n0000000b [label="{{ 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | { 1}}", shape=Mrecord, style=filled, fillcolor=green] >>>> + n0000000b:port1 -> n00000012 [style=bold] >>>> + n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow] >>>> + n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow] >>>> + n00000022 [label="{{ 0} | csi2-rx\n/dev/v4l-subdev4 | { 1}}", shape=Mrecord, style=filled, fillcolor=green] >>>> + n00000022:port1 -> n00000003:port0 >>>> + n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | { 0}}", shape=Mrecord, style=filled, fillcolor=green] >>>> + n00000027:port0 -> n00000022:port0 [style=bold] >>>> +} >>>> \ No newline at end of file >>>> diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst >>>> new file mode 100644 >>>> index 000000000000..33e63600ab2c >>>> --- /dev/null >>>> +++ b/Documentation/admin-guide/media/mali-c55.rst >>>> @@ -0,0 +1,330 @@ >>>> +.. SPDX-License-Identifier: GPL-2.0 >>>> + >>>> +========================================== >>>> +ARM Mali-C55 Image Signal Processor driver >>>> +========================================== >>>> + >>>> +Introduction >>>> +============ >>>> + >>>> +This file documents the driver for ARM's Mali-C55 Image Signal Processor. The >>>> +driver is located under drivers/media/platform/arm/mali-c55. >>>> + >>>> +The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from >>>> +sensors through either a parallel interface or a memory bus before processing it >>>> +and outputting it through an internal DMA engine. Two output pipelines are >>>> +possible (though one may not be fitted, depending on the implementation). These >>>> +are referred to as "Full resolution" and "Downscale", but the naming is historic >>>> +and both pipes are capable of cropping/scaling operations. The full resolution >>>> +pipe is also capable of outputting RAW data, bypassing much of the ISP's >>>> +processing. The downscale pipe cannot output RAW data. An integrated test >>>> +pattern generator can be used to drive the ISP and produce image data in the >>>> +absence of a connected camera sensor. The driver module is named mali_c55, and >>>> +is enabled through the CONFIG_VIDEO_MALI_C55 config option. >>>> + >>>> +The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and >>>> +expects camera sensors connected to the ISP to have V4L2 subdevice interfaces. >>>> + >>>> +Mali-C55 ISP hardware >>>> +===================== >>>> + >>>> +A high level functional view of the Mali-C55 ISP is presented below. The ISP >>>> +takes input from either a live source or through a DMA engine for memory input, >>>> +depending on the SoC integration.:: >>> This shows as >>> depending on the SoC integration.: >>> >>> in the generated output. >>> >>> Should you drop the '.' ? >> >> Yes, will do. >> >>>> + >>>> + +---------+ +----------+ +--------+ >>>> + | Sensor |--->| CSI-2 Rx | "Full Resolution" | DMA | >>>> + +---------+ +----------+ |\ Output +--->| Writer | >>>> + | | \ | +--------+ >>>> + | | \ +----------+ +------+---> Streaming I/O >>>> + +------------+ +------->| | | | | >>>> + | | | |-->| Mali-C55 |--+ >>>> + | DMA Reader |--------------->| | | ISP | | >>>> + | | | / | | | +---> Streaming I/O >>>> + +------------+ | / +----------+ | | >>>> + |/ +------+ >>>> + | +--------+ >>>> + +--->| DMA | >>>> + "Downscaled" | Writer | >>>> + Output +--------+ >>>> + >>>> +Media Controller Topology >>>> +========================= >>>> + >>>> +An example of the ISP's topology (as implemented in a system with an IMX415 >>>> +camera sensor and generic CSI-2 receiver) is below: >>>> + >>>> + >>>> +.. kernel-figure:: mali-c55-graph.dot >>>> + :alt: mali-c55-graph.dot >>>> + :align: center >>>> + >>>> +The driver has 4 V4L2 subdevices: >>>> + >>>> +- `mali_c55 isp`: Responsible for configuring input crop and color space >>>> + conversion >>>> +- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor. >>>> +- `mali_c55 resizer fr`: The Full-Resolution pipe resizer >>>> +- `mali_c55 resizer ds`: The Downscale pipe resizer >>>> + >>>> +The driver has 2 V4L2 video devices: >>>> + >>>> +- `mali-c55 fr`: The full-resolution pipe's capture device >>>> +- `mali-c55 ds`: The downscale pipe's capture device >>>> + >>>> +Frame sequences are synchronised across to two capture devices, meaning if one >>>> +pipe is started later than the other the sequence numbers returned in its >>>> +buffers will match those of the other pipe rather than starting from zero. >>>> + >>>> +Idiosyncrasies >>>> +-------------- >>>> + >>>> +**mali-c55 isp** >>> If the intention was to have a line break, this is not redendered in >>> the generated documentation. >>> >>>> +The `mali-c55 isp` subdevice has a single sink pad to which all sources of data >>>> +should be connected. The active source is selected by enabling the appropriate >>>> +media link and disabling all others. The ISP has two source pads, reflecting the >>>> +different paths through which it can internally route data. Tap points within >>>> +the ISP allow users to divert data to avoid processing by some or all of the >>>> +hardware's processing steps. The diagram below is intended only to highlight how >>>> +the bypassing works and is not a true reflection of those processing steps; for >>>> +a high-level functional block diagram see ARM's developer page for the >>>> +ISP [3]_:: >>>> + >>>> + +--------------------------------------------------------------+ >>>> + | Possible Internal ISP Data Routes | >>>> + | +------------+ +----------+ +------------+ | >>>> + +---+ | | | | | Colour | +---+ >>>> + | 0 |--+-->| Processing |->| Demosaic |->| Space |--->| 1 | >>>> + +---+ | | | | | | Conversion | +---+ >>>> + | | +------------+ +----------+ +------------+ | >>>> + | | +---+ >>>> + | +---------------------------------------------------| 2 | >>>> + | +---+ >>>> + | | >>>> + +--------------------------------------------------------------+ >>>> + >>>> + >>>> +.. flat-table:: >>>> + :header-rows: 1 >>>> + >>>> + * - Pad >>>> + - Direction >>>> + - Purpose >>>> + >>>> + * - 0 >>>> + - sink >>>> + - Data input, connected to the TPG and camera sensors >>>> + >>>> + * - 1 >>>> + - source >>>> + - RGB/YUV data, connected to the FR and DS V4L2 subdevices >>>> + >>>> + * - 2 >>>> + - source >>>> + - RAW bayer data, connected to the FR V4L2 subdevices >>>> + >>>> +The ISP is limited to both input and output resolutions between 640x480 and >>>> +8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt() >>>> +operations. >>>> + >>>> +**mali-c55 resizer fr** >>>> +The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different >>>> +insertion points in the hardware (either RAW or demosaiced data): >>>> + >>>> +.. flat-table:: >>>> + :header-rows: 1 >>>> + >>>> + * - Pad >>>> + - Direction >>>> + - Purpose >>>> + >>>> + * - 0 >>>> + - sink >>>> + - Data input connected to the ISP's demosaiced stream. >>>> + >>>> + * - 1 >>>> + - source >>>> + - Data output connected to the capture video device >>>> + >>>> + * - 2 >>>> + - sink >>>> + - Data input connected to the ISP's raw data stream >>>> + >>>> +The data source in use is selected through the routing API; two routes each of a >>>> +single stream are available: >>>> + >>>> +.. flat-table:: >>>> + :header-rows: 1 >>>> + >>>> + * - Sink Pad >>>> + - Source Pad >>>> + - Purpose >>>> + >>>> + * - 0 >>>> + - 1 >>>> + - Demosaiced data route >>>> + >>>> + * - 2 >>>> + - 1 >>>> + - Raw data route >>>> + >>>> + >>>> +If the demosaiced route is active then the FR pipe is only capable of output >>>> +in RGB/YUV formats. If the raw route is active then the output reflects the >>>> +input (which may be either Bayer or RGB/YUV data). >>>> + >>>> +Using the driver to capture video >>>> +================================= >>>> + >>>> +Using the media controller APIs we can configure the input source and ISP to >>>> +capture images in a variety of formats. In the examples below, configuring the >>>> +media graph is done with the v4l-utils [1]_ package's media-ctl utility. >>>> +Capturing the images is done with yavta [2]_. >>>> + >>>> +Configuring the input source >>>> +---------------------------- >>>> + >>>> +The first step is to set the input source that we wish by enabling the correct >>>> +media link. Using the example topology above, we can select the TPG as follows: >>>> + >>>> +.. code-block:: none >>>> + >>>> + media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]" >>>> + media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]" >>>> + >>>> +Capturing bayer data from the source and processing to RGB/YUV >>>> +-------------------------------------------------------------- >>>> +To capture 1920x1080 bayer data from the source and push it through the ISP's >>>> +full processing pipeline, we configure the data formats appropriately on the >>>> +source, ISP and resizer subdevices and set the FR resizer's routing to select >>>> +processed data. The media bus format on the resizer's source pad will be either >>>> +RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or >>>> +YUV. The ISP's debayering block outputs RGB data natively, setting the source >>>> +pad format to YUV10_1X30 enables the colour space conversion block. >>>> + >>>> +In this example we target RGB565 output, so select RGB121212_1X36 as the resizer >>>> +source pad's format: >>>> + >>>> +.. code-block:: none >>>> + >>>> + # Set formats on the TPG and ISP >>>> + media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]" >>>> + media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]" >>>> + media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]" >>> Shouldn't this be RGB121212_1X36 ? >> >> Oops, yes, obscured in testing by that being the sole supported format anyway. I'll fix it, thank you. >> >>>> + >>>> + # Set routing on the FR resizer >>>> + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]" >>>> + >>>> + # Set format on the resizer, must be done AFTER the routing. >>>> + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]" >>>> + >>>> +The downscale output can also be used to stream data at the same time. In this >>>> +case since only processed data can be captured through the downscale output no >>>> +routing need be set: >>>> + >>>> +.. code-block:: none >>>> + >>>> + # Set format on the resizer >>>> + media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]" >>>> + >>>> +Following which images can be captured from both the FR and DS output's video >>>> +devices (simultaneously, if desired): >>>> + >>>> +.. code-block:: none >>>> + >>>> + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 >>>> + yavta -f RGB565 -s 1920x1080 -c10 /dev/video1 >>>> + >>>> +Cropping the image >>>> +~~~~~~~~~~~~~~~~~~ >>>> + >>>> +Both the full resolution and downscale pipes can crop to a minimum resolution of >>>> +640x480. To crop the image simply configure the resizer's sink pad's crop and >>>> +compose rectangles and set the format on the video device: >>>> + >>>> +.. code-block:: none >>>> + >>>> + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]" >>>> + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" >>>> + yavta -f RGB565 -s 640x480 -c10 /dev/video0 >>>> + >>>> +Downscaling the image >>>> +~~~~~~~~~~~~~~~~~~~~~ >>>> + >>>> +Both the full resolution and downscale pipes can downscale the image by up to 8x >>>> +provided the minimum 640x480 resolution is adhered to. For the best image result >>>> +the scaling ratio for each dimension should be the same. To configure scaling we >>>> +use the compose rectangle on the resizer's sink pad: >>>> + >>>> +.. code-block:: none >>>> + >>>> + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]" >>>> + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" >>>> + yavta -f RGB565 -s 640x480 -c10 /dev/video0 >>>> + >>>> +Capturing images in YUV formats >>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> + >>>> +If we need to output YUV data rather than RGB the color space conversion block >>>> +needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the >>>> +resizer's source pad (the reduced bitdepth reflecting internal truncation after >>>> +color space conversion). We can then configure a capture format like NV12 (here >>>> +in its multi-planar variant) >>>> + >>>> +.. code-block:: none >>>> + >>>> + media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]" >>>> + yavta -f NV12M -s 1920x1080 -c10 /dev/video0 >>>> + >>>> +Capturing RGB data from the source and processing it with the resizers >>>> +---------------------------------------------------------------------- >>>> + >>>> +The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this >>>> +case although none of the image quality blocks would be used it can still >>>> +crop/scale the data in the usual way. >>>> + >>>> +To achieve this, the ISP's sink pad's format is set to >>>> +MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to >>>> +work with the ISP. Converting the camera sensor's output to that format is the >>>> +responsibility of external hardware. >>>> + >>>> +In this example we ask the test pattern generator to give us RGB data instead of >>>> +bayer. >>>> + >>>> +.. code-block:: none >>>> + >>>> + media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]" >>>> + media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]" >>>> + >>>> +Cropping or scaling the data can be done in exactly the same way as outlined >>>> +earlier. >>>> + >>>> +Capturing raw data from the source and outputting it unmodified >>>> +----------------------------------------------------------------- >>>> + >>>> +The ISP can additionally capture raw data from the source and output it on the >>>> +full resolution pipe only, completely unmodified. In this case the downscale >>>> +pipe can still process the data normally and be used at the same time. >>>> + >>>> +To configure raw bypass the FR resizer's subdevice's routing table needs to be >>>> +configured, followed by formats in the appropriate places: >>>> + >>>> +.. code-block:: none >>>> + >>>> + # We need to configure the routing table for the resizer to use the bypass >>>> + # path along with set formats on the resizer's bypass sink pad. Doing this >>>> + # necessitates a single media-ctl command, as multiple calls to the program >>>> + # reset the routing table. >>>> + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\ >>>> + -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\ >>>> + "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\ >>>> + "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]" >>>> + >>>> + # Set format on the video device and stream >>>> + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 >>> The example doesn't seem to show RAW formats though. >> >> A problem of terminology perhaps; I'm using the phrase to mean "unmodified >> source data" rather than "bayer formatted data". I can switch the example to >> use bayer data so its clearer? >> > I see.. Even if the paragraph mentions "raw data" and not "raw Bayer > data" I think showing how to capture Raw Bayer has the most value. > Then you can also specify it is possible to capture "unmodified source > data" and re-propose the above example. Sure - I'll re-work this bit to demonstrate Raw Bayer capture instead. > >>> I think it's also >>> relevant saying that the capture RAW data are expanded to 16 bits per >>> component with padding bits, regardless of the sensor's output >>> bitdepth. >> >> Yes good point - I'll add that in. >> >>> Overall this is a very nice documentation with a lot of use cases >>> demonstrated. >> Thanks! >>> Can we also say the ISP will be supported by libcamera :) ? >> >> Fine by me; though perhaps not until we get it merged there? >> > Well, it's a bit of a chicked&egg problem. I would add it right away. Righto - I'll add it in. > Thanks > j > >>> Thanks >>> j >>> >>>> + >>>> +References >>>> +========== >>>> +.. [1] https://git.linuxtv.org/v4l-utils.git/ >>>> +.. [2] https://git.ideasonboard.org/yavta.git >>>> +.. [3] https://developer.arm.com/Processors/Mali-C55 >>>> diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst >>>> index f4bb2605f07e..af033c892808 100644 >>>> --- a/Documentation/admin-guide/media/v4l-drivers.rst >>>> +++ b/Documentation/admin-guide/media/v4l-drivers.rst >>>> @@ -17,6 +17,7 @@ Video4Linux (V4L) driver-specific documentation >>>> imx7 >>>> ipu3 >>>> ivtv >>>> + mali-c55 >>>> mgb4 >>>> omap3isp >>>> omap4_camera >>>> -- >>>> 2.34.1 >>>>