[PATCH v5 0/4] media: raspberrypi: Support RPi5's CFE

[PATCH v5 0/4] media: raspberrypi: Support RPi5's CFE

[Tomi Valkeinen]

This series adds support to the CFE hardware block on RaspberryPi 5. The
CFE (Camera Front End) contains a CSI-2 receiver and Front End, a small
ISP.

To run this, you need the basic RPi5 kernel support plus relevant dts
changes to enable the cfe and camera. My work branch with everything
needed to run CFE can be found from:

git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git rp1-cfe

A few notes about the patches:

- The original work was done by RaspberryPi, mostly by Naushir Patuck.
- The second video node only sets V4L2_CAP_META_CAPTURE instead of both
  V4L2_CAP_META_CAPTURE and V4L2_CAP_META_CAPTURE like the other nodes.
  This is a temporary workaround for userspace (libcamera), and
  hopefully can be removed soon.

I have tested this with:
- A single IMX219 sensor connected to the RPi5's CSI-2 port
- Arducam's UB960 FPD-Link board with four imx219 sensors connected

 Tomi

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
Changes in v5:
- Add "depends on PM". The platforms that use CFE will always have PM in
  practice, and it's not worth supporting both the PM and !PM cases as
  it adds complexity to the driver.
- Link to v4: https://lore.kernel.org/r/20240904-rp1-cfe-v4-0-f1b5b3d69c81@ideasonboard.com

Changes in v4:
- Drop unnecessary clock-lanes from the DT bindings
- Drop unnecessary linux-media from MAINTAINERS entry
- Drop unnecessary conversion to bool with !!
- Don't set cap->bus_info in cfe_querycap()
- Make debugfs files not readable by the world
- Check the return value of v4l2_fwnode_endpoint_parse()
- Remove the code dealing with remote_ep_fwnode. Instead use
  v4l2_create_fwnode_links_to_pad() and media_pad_remote_pad_unique() to
  create the link and get the pad index.
- Add cfe/csi2/fe/dphy argument to the respective dbg/info/err print
  macros.
- Drop some debug prints and add a few, clarifying the prints for
  enabling and disabling the streams.
- Some cosmetic changes (linefeed, drop unnecessary assignment, move a
  define)
- Link to v3: https://lore.kernel.org/r/20240815-rp1-cfe-v3-0-e15a979db327@ideasonboard.com

Changes in v3:
- Based on v6.11-rc3. The PiSP BE series is now in upstream so no extra
  dependencies are needed.
- Fixed cfe_remove() return value, as the .remove hook has changed
- Added Krzysztof's Rb.
- Link to v2: https://lore.kernel.org/r/20240620-rp1-cfe-v2-0-b8b48fdba3b3@ideasonboard.com

Changes in v2:
- Change the compatible string back to raspberrypi,rp1-cfe from raspberrypi,rpi5-rp1-cfe
- Drop the references to rp1 headers in the DT binding example. This
  allows compiling the example without the rp1 support.
- Fix missing remap lines for mono formats
- Fix csi2_pad_set_fmt() so that the format can be changed back to the
  sink's format from 16-bit or compressed format.
- Link to v1: https://lore.kernel.org/r/20240318-rp1-cfe-v1-0-ac6d960ff22d@ideasonboard.com

---
Tomi Valkeinen (4):
      media: uapi: Add meta formats for PiSP FE config and stats
      dt-bindings: media: Add bindings for raspberrypi,rp1-cfe
      media: raspberrypi: Add support for RP1-CFE
      media: admin-guide: Document the Raspberry Pi CFE (rp1-cfe)

 .../admin-guide/media/raspberrypi-rp1-cfe.dot      |   27 +
 .../admin-guide/media/raspberrypi-rp1-cfe.rst      |   78 +
 Documentation/admin-guide/media/v4l-drivers.rst    |    1 +
 .../bindings/media/raspberrypi,rp1-cfe.yaml        |   93 +
 .../userspace-api/media/v4l/meta-formats.rst       |    1 +
 .../userspace-api/media/v4l/metafmt-pisp-fe.rst    |   39 +
 MAINTAINERS                                        |    7 +
 drivers/media/platform/raspberrypi/Kconfig         |    1 +
 drivers/media/platform/raspberrypi/Makefile        |    1 +
 drivers/media/platform/raspberrypi/rp1-cfe/Kconfig |   15 +
 .../media/platform/raspberrypi/rp1-cfe/Makefile    |    6 +
 .../media/platform/raspberrypi/rp1-cfe/cfe-fmts.h  |  332 +++
 .../media/platform/raspberrypi/rp1-cfe/cfe-trace.h |  196 ++
 drivers/media/platform/raspberrypi/rp1-cfe/cfe.c   | 2487 ++++++++++++++++++++
 drivers/media/platform/raspberrypi/rp1-cfe/cfe.h   |   43 +
 drivers/media/platform/raspberrypi/rp1-cfe/csi2.c  |  583 +++++
 drivers/media/platform/raspberrypi/rp1-cfe/csi2.h  |   89 +
 drivers/media/platform/raspberrypi/rp1-cfe/dphy.c  |  180 ++
 drivers/media/platform/raspberrypi/rp1-cfe/dphy.h  |   27 +
 .../media/platform/raspberrypi/rp1-cfe/pisp-fe.c   |  581 +++++
 .../media/platform/raspberrypi/rp1-cfe/pisp-fe.h   |   53 +
 drivers/media/v4l2-core/v4l2-ioctl.c               |    2 +
 .../uapi/linux/media/raspberrypi/pisp_fe_config.h  |  273 +++
 .../linux/media/raspberrypi/pisp_fe_statistics.h   |   64 +
 include/uapi/linux/videodev2.h                     |    2 +
 25 files changed, 5181 insertions(+)
---
base-commit: 431c1646e1f86b949fa3685efc50b660a364c2b6
change-id: 20240314-rp1-cfe-142b628b7214

Best regards,
-- 
Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

[PATCH v5 1/4] media: uapi: Add meta formats for PiSP FE config and stats

[Tomi Valkeinen]

Add two meta formats for PiSP FE: V4L2_META_FMT_RPI_FE_CFG and
V4L2_META_FMT_RPI_FE_STATS. The former is used to provide configuration
for the FE and the latter is used to read the statistics from the FE.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../userspace-api/media/v4l/meta-formats.rst       |  1 +
 .../userspace-api/media/v4l/metafmt-pisp-fe.rst    | 39 ++++++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c               |  2 ++
 include/uapi/linux/videodev2.h                     |  2 ++
 4 files changed, 44 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index c6e56b5888bc..86ffb3bc8ade 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -16,6 +16,7 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-generic
     metafmt-intel-ipu3
     metafmt-pisp-be
+    metafmt-pisp-fe
     metafmt-rkisp1
     metafmt-uvc
     metafmt-vivid
diff --git a/Documentation/userspace-api/media/v4l/metafmt-pisp-fe.rst b/Documentation/userspace-api/media/v4l/metafmt-pisp-fe.rst
new file mode 100644
index 000000000000..fddeada83e4a
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/metafmt-pisp-fe.rst
@@ -0,0 +1,39 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _v4l2-meta-fmt-rpi-fe-cfg:
+
+************************
+V4L2_META_FMT_RPI_FE_CFG
+************************
+
+Raspberry Pi PiSP Front End configuration format
+================================================
+
+The Raspberry Pi PiSP Front End image signal processor is configured by
+userspace by providing a buffer of configuration parameters to the
+`rp1-cfe-fe-config` output video device node using the
+:c:type:`v4l2_meta_format` interface.
+
+The `Raspberry Pi PiSP technical specification
+<https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf>`_
+provide detailed description of the Front End configuration and programming
+model.
+
+.. _v4l2-meta-fmt-rpi-fe-stats:
+
+**************************
+V4L2_META_FMT_RPI_FE_STATS
+**************************
+
+Raspberry Pi PiSP Front End statistics format
+=============================================
+
+The Raspberry Pi PiSP Front End image signal processor provides statistics data
+by writing to a buffer provided via the `rp1-cfe-fe-stats` capture video device
+node using the
+:c:type:`v4l2_meta_format` interface.
+
+The `Raspberry Pi PiSP technical specification
+<https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf>`_
+provide detailed description of the Front End configuration and programming
+model.
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 5eb4d797d259..95c2c4a97966 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1466,6 +1466,8 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
 	case V4L2_META_FMT_RPI_BE_CFG:	descr = "RPi PiSP BE Config format"; break;
+	case V4L2_META_FMT_RPI_FE_CFG:  descr = "RPi PiSP FE Config format"; break;
+	case V4L2_META_FMT_RPI_FE_STATS: descr = "RPi PiSP FE Statistics format"; break;
 	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
 	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8-bit Generic Meta, 10b CSI-2"; break;
 	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8-bit Generic Meta, 12b CSI-2"; break;
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 4e91362da6da..9f68cd55248f 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -857,6 +857,8 @@ struct v4l2_pix_format {
 
 /* Vendor specific - used for RaspberryPi PiSP */
 #define V4L2_META_FMT_RPI_BE_CFG	v4l2_fourcc('R', 'P', 'B', 'C') /* PiSP BE configuration */
+#define V4L2_META_FMT_RPI_FE_CFG	v4l2_fourcc('R', 'P', 'F', 'C') /* PiSP FE configuration */
+#define V4L2_META_FMT_RPI_FE_STATS	v4l2_fourcc('R', 'P', 'F', 'S') /* PiSP FE stats */
 
 #ifdef __KERNEL__
 /*

-- 
2.43.0

[PATCH v5 2/4] dt-bindings: media: Add bindings for raspberrypi,rp1-cfe

[Tomi Valkeinen]

Add DT bindings for raspberrypi,rp1-cfe.

Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../bindings/media/raspberrypi,rp1-cfe.yaml        | 93 ++++++++++++++++++++++
 1 file changed, 93 insertions(+)

diff --git a/Documentation/devicetree/bindings/media/raspberrypi,rp1-cfe.yaml b/Documentation/devicetree/bindings/media/raspberrypi,rp1-cfe.yaml
new file mode 100644
index 000000000000..eba5394719b9
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/raspberrypi,rp1-cfe.yaml
@@ -0,0 +1,93 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/media/raspberrypi,rp1-cfe.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Raspberry Pi PiSP Camera Front End
+
+maintainers:
+  - Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
+  - Raspberry Pi Kernel Maintenance <kernel-list@raspberrypi.com>
+
+description: |
+  The Raspberry Pi PiSP Camera Front End is a module in Raspberrypi 5's RP1 I/O
+  controller, that contains:
+  - MIPI D-PHY
+  - MIPI CSI-2 receiver
+  - Simple image processor (called PiSP Front End, or FE)
+
+  The FE documentation is available at:
+  https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf
+
+  The PHY and CSI-2 receiver part have no public documentation.
+
+properties:
+  compatible:
+    items:
+      - const: raspberrypi,rp1-cfe
+
+  reg:
+    items:
+      - description: CSI-2 registers
+      - description: D-PHY registers
+      - description: MIPI CFG (a simple top-level mux) registers
+      - description: FE registers
+
+  interrupts:
+    maxItems: 1
+
+  clocks:
+    maxItems: 1
+
+  port:
+    $ref: /schemas/graph.yaml#/$defs/port-base
+    additionalProperties: false
+    description: CSI-2 RX Port
+
+    properties:
+      endpoint:
+        $ref: video-interfaces.yaml#
+        unevaluatedProperties: false
+
+        properties:
+          data-lanes:
+            minItems: 1
+            maxItems: 4
+
+        required:
+          - data-lanes
+
+required:
+  - compatible
+  - reg
+  - interrupts
+  - clocks
+
+additionalProperties: false
+
+examples:
+  - |
+    rp1 {
+      #address-cells = <2>;
+      #size-cells = <2>;
+
+      csi@110000 {
+        compatible = "raspberrypi,rp1-cfe";
+        reg = <0xc0 0x40110000 0x0 0x100>,
+              <0xc0 0x40114000 0x0 0x100>,
+              <0xc0 0x40120000 0x0 0x100>,
+              <0xc0 0x40124000 0x0 0x1000>;
+
+        interrupts = <42>;
+
+        clocks = <&rp1_clocks>;
+
+        port {
+          csi_ep: endpoint {
+            remote-endpoint = <&cam_endpoint>;
+            data-lanes = <1 2>;
+          };
+        };
+      };
+    };

-- 
2.43.0

[PATCH v5 4/4] media: admin-guide: Document the Raspberry Pi CFE (rp1-cfe)

[Tomi Valkeinen]

Add documentation for rp1-cfe driver.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../admin-guide/media/raspberrypi-rp1-cfe.dot      | 27 ++++++++
 .../admin-guide/media/raspberrypi-rp1-cfe.rst      | 78 ++++++++++++++++++++++
 Documentation/admin-guide/media/v4l-drivers.rst    |  1 +
 3 files changed, 106 insertions(+)

diff --git a/Documentation/admin-guide/media/raspberrypi-rp1-cfe.dot b/Documentation/admin-guide/media/raspberrypi-rp1-cfe.dot
new file mode 100644
index 000000000000..7717f2291049
--- /dev/null
+++ b/Documentation/admin-guide/media/raspberrypi-rp1-cfe.dot
@@ -0,0 +1,27 @@
+digraph board {
+	rankdir=TB
+	n00000001 [label="{{<port0> 0} | csi2\n/dev/v4l-subdev0 | {<port1> 1 | <port2> 2 | <port3> 3 | <port4> 4}}", shape=Mrecord, style=filled, fillcolor=green]
+	n00000001:port1 -> n00000011 [style=dashed]
+	n00000001:port1 -> n00000007:port0
+	n00000001:port2 -> n00000015
+	n00000001:port2 -> n00000007:port0 [style=dashed]
+	n00000001:port3 -> n00000019 [style=dashed]
+	n00000001:port3 -> n00000007:port0 [style=dashed]
+	n00000001:port4 -> n0000001d [style=dashed]
+	n00000001:port4 -> n00000007:port0 [style=dashed]
+	n00000007 [label="{{<port0> 0 | <port1> 1} | pisp-fe\n/dev/v4l-subdev1 | {<port2> 2 | <port3> 3 | <port4> 4}}", shape=Mrecord, style=filled, fillcolor=green]
+	n00000007:port2 -> n00000021
+	n00000007:port3 -> n00000025 [style=dashed]
+	n00000007:port4 -> n00000029
+	n0000000d [label="{imx219 6-0010\n/dev/v4l-subdev2 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
+	n0000000d:port0 -> n00000001:port0 [style=bold]
+	n00000011 [label="rp1-cfe-csi2-ch0\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
+	n00000015 [label="rp1-cfe-csi2-ch1\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
+	n00000019 [label="rp1-cfe-csi2-ch2\n/dev/video2", shape=box, style=filled, fillcolor=yellow]
+	n0000001d [label="rp1-cfe-csi2-ch3\n/dev/video3", shape=box, style=filled, fillcolor=yellow]
+	n00000021 [label="rp1-cfe-fe-image0\n/dev/video4", shape=box, style=filled, fillcolor=yellow]
+	n00000025 [label="rp1-cfe-fe-image1\n/dev/video5", shape=box, style=filled, fillcolor=yellow]
+	n00000029 [label="rp1-cfe-fe-stats\n/dev/video6", shape=box, style=filled, fillcolor=yellow]
+	n0000002d [label="rp1-cfe-fe-config\n/dev/video7", shape=box, style=filled, fillcolor=yellow]
+	n0000002d -> n00000007:port1
+}
diff --git a/Documentation/admin-guide/media/raspberrypi-rp1-cfe.rst b/Documentation/admin-guide/media/raspberrypi-rp1-cfe.rst
new file mode 100644
index 000000000000..668d978a9875
--- /dev/null
+++ b/Documentation/admin-guide/media/raspberrypi-rp1-cfe.rst
@@ -0,0 +1,78 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+============================================
+Raspberry Pi PiSP Camera Front End (rp1-cfe)
+============================================
+
+The PiSP Camera Front End
+=========================
+
+The PiSP Camera Front End (CFE) is a module which combines a CSI-2 receiver with
+a simple ISP, called the Front End (FE).
+
+The CFE has four DMA engines and can write frames from four separate streams
+received from the CSI-2 to the memory. One of those streams can also be routed
+directly to the FE, which can do minimal image processing, write two versions
+(e.g. non-scaled and downscaled versions) of the received frames to memory and
+provide statistics of the received frames.
+
+The FE registers are documented in the `Raspberry Pi Image Signal Processor
+(ISP) Specification document
+<https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf>`_,
+and example code for FE can be found in `libpisp
+<https://github.com/raspberrypi/libpisp>`_.
+
+The rp1-cfe driver
+==================
+
+The Raspberry Pi PiSP Camera Front End (rp1-cfe) driver is located under
+drivers/media/platform/raspberrypi/rp1-cfe. It uses the `V4L2 API` to register
+a number of video capture and output devices, the `V4L2 subdev API` to register
+subdevices for the CSI-2 received and the FE that connects the video devices in
+a single media graph realized using the `Media Controller (MC) API`.
+
+The media topology registered by the `rp1-cfe` driver, in this particular
+example connected to an imx219 sensor, is the following one:
+
+.. _rp1-cfe-topology:
+
+.. kernel-figure:: raspberrypi-rp1-cfe.dot
+    :alt:   Diagram of an example media pipeline topology
+    :align: center
+
+The media graph contains the following video device nodes:
+
+- rp1-cfe-csi2-ch0: capture device for the first CSI-2 stream
+- rp1-cfe-csi2-ch1: capture device for the second CSI-2 stream
+- rp1-cfe-csi2-ch2: capture device for the third CSI-2 stream
+- rp1-cfe-csi2-ch3: capture device for the fourth CSI-2 stream
+- rp1-cfe-fe-image0: capture device for the first FE output
+- rp1-cfe-fe-image1: capture device for the second FE output
+- rp1-cfe-fe-stats: capture device for the FE statistics
+- rp1-cfe-fe-config: output device for FE configuration
+
+rp1-cfe-csi2-chX
+----------------
+
+The rp1-cfe-csi2-chX capture devices are normal V4L2 capture devices which
+can be used to capture video frames or metadata received from the CSI-2.
+
+rp1-cfe-fe-image0, rp1-cfe-fe-image1
+------------------------------------
+
+The rp1-cfe-fe-image0 and rp1-cfe-fe-image1 capture devices are used to write
+the processed frames to memory.
+
+rp1-cfe-fe-stats
+----------------
+
+The format of the FE statistics buffer is defined by
+:c:type:`pisp_statistics` C structure and the meaning of each parameter is
+described in the `PiSP specification` document.
+
+rp1-cfe-fe-config
+-----------------
+
+The format of the FE configuration buffer is defined by
+:c:type:`pisp_fe_config` C structure and the meaning of each parameter is
+described in the `PiSP specification` document.
diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst
index b6af448b9fe9..61da154e079a 100644
--- a/Documentation/admin-guide/media/v4l-drivers.rst
+++ b/Documentation/admin-guide/media/v4l-drivers.rst
@@ -26,6 +26,7 @@ Video4Linux (V4L) driver-specific documentation
 	raspberrypi-pisp-be
 	rcar-fdp1
 	rkisp1
+	raspberrypi-rp1-cfe
 	saa7134
 	si470x
 	si4713

-- 
2.43.0

Re: [PATCH v5 0/4] media: raspberrypi: Support RPi5's CFE

[Tomi Valkeinen]

Hi,

On 10/09/2024 11:07, Tomi Valkeinen wrote:
> This series adds support to the CFE hardware block on RaspberryPi 5. The
> CFE (Camera Front End) contains a CSI-2 receiver and Front End, a small
> ISP.
> 
> To run this, you need the basic RPi5 kernel support plus relevant dts
> changes to enable the cfe and camera. My work branch with everything
> needed to run CFE can be found from:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git rp1-cfe
> 
> A few notes about the patches:
> 
> - The original work was done by RaspberryPi, mostly by Naushir Patuck.
> - The second video node only sets V4L2_CAP_META_CAPTURE instead of both
>    V4L2_CAP_META_CAPTURE and V4L2_CAP_META_CAPTURE like the other nodes.
>    This is a temporary workaround for userspace (libcamera), and
>    hopefully can be removed soon.
> 
> I have tested this with:
> - A single IMX219 sensor connected to the RPi5's CSI-2 port
> - Arducam's UB960 FPD-Link board with four imx219 sensors connected
> 
>   Tomi
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
> Changes in v5:
> - Add "depends on PM". The platforms that use CFE will always have PM in
>    practice, and it's not worth supporting both the PM and !PM cases as
>    it adds complexity to the driver.
> - Link to v4: https://lore.kernel.org/r/20240904-rp1-cfe-v4-0-f1b5b3d69c81@ideasonboard.com

Is this solution to the PM issue ok for everyone? It feels most sensible 
to me. Any other comments?

  Tomi

> Changes in v4:
> - Drop unnecessary clock-lanes from the DT bindings
> - Drop unnecessary linux-media from MAINTAINERS entry
> - Drop unnecessary conversion to bool with !!
> - Don't set cap->bus_info in cfe_querycap()
> - Make debugfs files not readable by the world
> - Check the return value of v4l2_fwnode_endpoint_parse()
> - Remove the code dealing with remote_ep_fwnode. Instead use
>    v4l2_create_fwnode_links_to_pad() and media_pad_remote_pad_unique() to
>    create the link and get the pad index.
> - Add cfe/csi2/fe/dphy argument to the respective dbg/info/err print
>    macros.
> - Drop some debug prints and add a few, clarifying the prints for
>    enabling and disabling the streams.
> - Some cosmetic changes (linefeed, drop unnecessary assignment, move a
>    define)
> - Link to v3: https://lore.kernel.org/r/20240815-rp1-cfe-v3-0-e15a979db327@ideasonboard.com
> 
> Changes in v3:
> - Based on v6.11-rc3. The PiSP BE series is now in upstream so no extra
>    dependencies are needed.
> - Fixed cfe_remove() return value, as the .remove hook has changed
> - Added Krzysztof's Rb.
> - Link to v2: https://lore.kernel.org/r/20240620-rp1-cfe-v2-0-b8b48fdba3b3@ideasonboard.com
> 
> Changes in v2:
> - Change the compatible string back to raspberrypi,rp1-cfe from raspberrypi,rpi5-rp1-cfe
> - Drop the references to rp1 headers in the DT binding example. This
>    allows compiling the example without the rp1 support.
> - Fix missing remap lines for mono formats
> - Fix csi2_pad_set_fmt() so that the format can be changed back to the
>    sink's format from 16-bit or compressed format.
> - Link to v1: https://lore.kernel.org/r/20240318-rp1-cfe-v1-0-ac6d960ff22d@ideasonboard.com
> 
> ---
> Tomi Valkeinen (4):
>        media: uapi: Add meta formats for PiSP FE config and stats
>        dt-bindings: media: Add bindings for raspberrypi,rp1-cfe
>        media: raspberrypi: Add support for RP1-CFE
>        media: admin-guide: Document the Raspberry Pi CFE (rp1-cfe)
> 
>   .../admin-guide/media/raspberrypi-rp1-cfe.dot      |   27 +
>   .../admin-guide/media/raspberrypi-rp1-cfe.rst      |   78 +
>   Documentation/admin-guide/media/v4l-drivers.rst    |    1 +
>   .../bindings/media/raspberrypi,rp1-cfe.yaml        |   93 +
>   .../userspace-api/media/v4l/meta-formats.rst       |    1 +
>   .../userspace-api/media/v4l/metafmt-pisp-fe.rst    |   39 +
>   MAINTAINERS                                        |    7 +
>   drivers/media/platform/raspberrypi/Kconfig         |    1 +
>   drivers/media/platform/raspberrypi/Makefile        |    1 +
>   drivers/media/platform/raspberrypi/rp1-cfe/Kconfig |   15 +
>   .../media/platform/raspberrypi/rp1-cfe/Makefile    |    6 +
>   .../media/platform/raspberrypi/rp1-cfe/cfe-fmts.h  |  332 +++
>   .../media/platform/raspberrypi/rp1-cfe/cfe-trace.h |  196 ++
>   drivers/media/platform/raspberrypi/rp1-cfe/cfe.c   | 2487 ++++++++++++++++++++
>   drivers/media/platform/raspberrypi/rp1-cfe/cfe.h   |   43 +
>   drivers/media/platform/raspberrypi/rp1-cfe/csi2.c  |  583 +++++
>   drivers/media/platform/raspberrypi/rp1-cfe/csi2.h  |   89 +
>   drivers/media/platform/raspberrypi/rp1-cfe/dphy.c  |  180 ++
>   drivers/media/platform/raspberrypi/rp1-cfe/dphy.h  |   27 +
>   .../media/platform/raspberrypi/rp1-cfe/pisp-fe.c   |  581 +++++
>   .../media/platform/raspberrypi/rp1-cfe/pisp-fe.h   |   53 +
>   drivers/media/v4l2-core/v4l2-ioctl.c               |    2 +
>   .../uapi/linux/media/raspberrypi/pisp_fe_config.h  |  273 +++
>   .../linux/media/raspberrypi/pisp_fe_statistics.h   |   64 +
>   include/uapi/linux/videodev2.h                     |    2 +
>   25 files changed, 5181 insertions(+)
> ---
> base-commit: 431c1646e1f86b949fa3685efc50b660a364c2b6
> change-id: 20240314-rp1-cfe-142b628b7214
> 
> Best regards,

Re: [PATCH v5 0/4] media: raspberrypi: Support RPi5's CFE

[Jacopo Mondi]

Hi Tomi

On Thu, Sep 26, 2024 at 10:13:29AM GMT, Tomi Valkeinen wrote:
> Hi,
>
> On 10/09/2024 11:07, Tomi Valkeinen wrote:
> > This series adds support to the CFE hardware block on RaspberryPi 5. The
> > CFE (Camera Front End) contains a CSI-2 receiver and Front End, a small
> > ISP.
> >
> > To run this, you need the basic RPi5 kernel support plus relevant dts
> > changes to enable the cfe and camera. My work branch with everything
> > needed to run CFE can be found from:
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git rp1-cfe
> >
> > A few notes about the patches:
> >
> > - The original work was done by RaspberryPi, mostly by Naushir Patuck.
> > - The second video node only sets V4L2_CAP_META_CAPTURE instead of both
> >    V4L2_CAP_META_CAPTURE and V4L2_CAP_META_CAPTURE like the other nodes.
> >    This is a temporary workaround for userspace (libcamera), and
> >    hopefully can be removed soon.
> >
> > I have tested this with:
> > - A single IMX219 sensor connected to the RPi5's CSI-2 port
> > - Arducam's UB960 FPD-Link board with four imx219 sensors connected
> >
> >   Tomi
> >
> > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > ---
> > Changes in v5:
> > - Add "depends on PM". The platforms that use CFE will always have PM in
> >    practice, and it's not worth supporting both the PM and !PM cases as
> >    it adds complexity to the driver.
> > - Link to v4: https://lore.kernel.org/r/20240904-rp1-cfe-v4-0-f1b5b3d69c81@ideasonboard.com
>
> Is this solution to the PM issue ok for everyone? It feels most sensible to
> me. Any other comments?

There was consensus at the media summit that platform drivers are
allowed to depend on PM.

I'll do the same for my latest PiSP BE series.

>
>  Tomi
>
> > Changes in v4:
> > - Drop unnecessary clock-lanes from the DT bindings
> > - Drop unnecessary linux-media from MAINTAINERS entry
> > - Drop unnecessary conversion to bool with !!
> > - Don't set cap->bus_info in cfe_querycap()
> > - Make debugfs files not readable by the world
> > - Check the return value of v4l2_fwnode_endpoint_parse()
> > - Remove the code dealing with remote_ep_fwnode. Instead use
> >    v4l2_create_fwnode_links_to_pad() and media_pad_remote_pad_unique() to
> >    create the link and get the pad index.
> > - Add cfe/csi2/fe/dphy argument to the respective dbg/info/err print
> >    macros.
> > - Drop some debug prints and add a few, clarifying the prints for
> >    enabling and disabling the streams.
> > - Some cosmetic changes (linefeed, drop unnecessary assignment, move a
> >    define)
> > - Link to v3: https://lore.kernel.org/r/20240815-rp1-cfe-v3-0-e15a979db327@ideasonboard.com
> >
> > Changes in v3:
> > - Based on v6.11-rc3. The PiSP BE series is now in upstream so no extra
> >    dependencies are needed.
> > - Fixed cfe_remove() return value, as the .remove hook has changed
> > - Added Krzysztof's Rb.
> > - Link to v2: https://lore.kernel.org/r/20240620-rp1-cfe-v2-0-b8b48fdba3b3@ideasonboard.com
> >
> > Changes in v2:
> > - Change the compatible string back to raspberrypi,rp1-cfe from raspberrypi,rpi5-rp1-cfe
> > - Drop the references to rp1 headers in the DT binding example. This
> >    allows compiling the example without the rp1 support.
> > - Fix missing remap lines for mono formats
> > - Fix csi2_pad_set_fmt() so that the format can be changed back to the
> >    sink's format from 16-bit or compressed format.
> > - Link to v1: https://lore.kernel.org/r/20240318-rp1-cfe-v1-0-ac6d960ff22d@ideasonboard.com
> >
> > ---
> > Tomi Valkeinen (4):
> >        media: uapi: Add meta formats for PiSP FE config and stats
> >        dt-bindings: media: Add bindings for raspberrypi,rp1-cfe
> >        media: raspberrypi: Add support for RP1-CFE
> >        media: admin-guide: Document the Raspberry Pi CFE (rp1-cfe)
> >
> >   .../admin-guide/media/raspberrypi-rp1-cfe.dot      |   27 +
> >   .../admin-guide/media/raspberrypi-rp1-cfe.rst      |   78 +
> >   Documentation/admin-guide/media/v4l-drivers.rst    |    1 +
> >   .../bindings/media/raspberrypi,rp1-cfe.yaml        |   93 +
> >   .../userspace-api/media/v4l/meta-formats.rst       |    1 +
> >   .../userspace-api/media/v4l/metafmt-pisp-fe.rst    |   39 +
> >   MAINTAINERS                                        |    7 +
> >   drivers/media/platform/raspberrypi/Kconfig         |    1 +
> >   drivers/media/platform/raspberrypi/Makefile        |    1 +
> >   drivers/media/platform/raspberrypi/rp1-cfe/Kconfig |   15 +
> >   .../media/platform/raspberrypi/rp1-cfe/Makefile    |    6 +
> >   .../media/platform/raspberrypi/rp1-cfe/cfe-fmts.h  |  332 +++
> >   .../media/platform/raspberrypi/rp1-cfe/cfe-trace.h |  196 ++
> >   drivers/media/platform/raspberrypi/rp1-cfe/cfe.c   | 2487 ++++++++++++++++++++
> >   drivers/media/platform/raspberrypi/rp1-cfe/cfe.h   |   43 +
> >   drivers/media/platform/raspberrypi/rp1-cfe/csi2.c  |  583 +++++
> >   drivers/media/platform/raspberrypi/rp1-cfe/csi2.h  |   89 +
> >   drivers/media/platform/raspberrypi/rp1-cfe/dphy.c  |  180 ++
> >   drivers/media/platform/raspberrypi/rp1-cfe/dphy.h  |   27 +
> >   .../media/platform/raspberrypi/rp1-cfe/pisp-fe.c   |  581 +++++
> >   .../media/platform/raspberrypi/rp1-cfe/pisp-fe.h   |   53 +
> >   drivers/media/v4l2-core/v4l2-ioctl.c               |    2 +
> >   .../uapi/linux/media/raspberrypi/pisp_fe_config.h  |  273 +++
> >   .../linux/media/raspberrypi/pisp_fe_statistics.h   |   64 +
> >   include/uapi/linux/videodev2.h                     |    2 +
> >   25 files changed, 5181 insertions(+)
> > ---
> > base-commit: 431c1646e1f86b949fa3685efc50b660a364c2b6
> > change-id: 20240314-rp1-cfe-142b628b7214
> >
> > Best regards,
>
>

Re: [PATCH v5 0/4] media: raspberrypi: Support RPi5's CFE

[Sakari Ailus]

Hi Jacopo,

On Thu, Sep 26, 2024 at 11:04:36AM +0200, Jacopo Mondi wrote:
> Hi Tomi
> 
> On Thu, Sep 26, 2024 at 10:13:29AM GMT, Tomi Valkeinen wrote:
> > Hi,
> >
> > On 10/09/2024 11:07, Tomi Valkeinen wrote:
> > > This series adds support to the CFE hardware block on RaspberryPi 5. The
> > > CFE (Camera Front End) contains a CSI-2 receiver and Front End, a small
> > > ISP.
> > >
> > > To run this, you need the basic RPi5 kernel support plus relevant dts
> > > changes to enable the cfe and camera. My work branch with everything
> > > needed to run CFE can be found from:
> > >
> > > git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git rp1-cfe
> > >
> > > A few notes about the patches:
> > >
> > > - The original work was done by RaspberryPi, mostly by Naushir Patuck.
> > > - The second video node only sets V4L2_CAP_META_CAPTURE instead of both
> > >    V4L2_CAP_META_CAPTURE and V4L2_CAP_META_CAPTURE like the other nodes.
> > >    This is a temporary workaround for userspace (libcamera), and
> > >    hopefully can be removed soon.
> > >
> > > I have tested this with:
> > > - A single IMX219 sensor connected to the RPi5's CSI-2 port
> > > - Arducam's UB960 FPD-Link board with four imx219 sensors connected
> > >
> > >   Tomi
> > >
> > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > ---
> > > Changes in v5:
> > > - Add "depends on PM". The platforms that use CFE will always have PM in
> > >    practice, and it's not worth supporting both the PM and !PM cases as
> > >    it adds complexity to the driver.
> > > - Link to v4: https://lore.kernel.org/r/20240904-rp1-cfe-v4-0-f1b5b3d69c81@ideasonboard.com
> >
> > Is this solution to the PM issue ok for everyone? It feels most sensible to
> > me. Any other comments?
> 
> There was consensus at the media summit that platform drivers are
> allowed to depend on PM.

I'd say you should have generally a good idea these devices aren't found in
systems without runtime PM support and that mostly applies to platform
drivers.

> 
> I'll do the same for my latest PiSP BE series.

Ack.

-- 
Kind regards,

Sakari Ailus