[PATCH v3 0/4] media: raspberrypi: Support RPi5's CFE

[PATCH v3 0/4] media: raspberrypi: Support RPi5's CFE

[Tomi Valkeinen]

This series adds support to the CFE hardware block on RaspberryPi 5. The
CFE (Camera Front End) contains a CSI-2 receiver and Front End, a small
ISP.

To run this, you need the basic RPi5 kernel support plus relevant dts
changes to enable the cfe and camera. My work branch with everything
needed to run CFE can be found from:

git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git rp1-cfe

A few notes about the patches:

- The original work was done by RaspberryPi, mostly by Naushir Patuck.
- The second video node only sets V4L2_CAP_META_CAPTURE instead of both
  V4L2_CAP_META_CAPTURE and V4L2_CAP_META_CAPTURE like the other nodes.
  This is a temporary workaround for userspace (libcamera), and
  hopefully can be removed soon.

I have tested this with:
- A single IMX219 sensor connected to the RPi5's CSI-2 port
- Arducam's UB960 FPD-Link board with four imx219 sensors connected

 Tomi

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
Changes in v3:
- Based on v6.11-rc3. The PiSP BE series is now in upstream so no extra
  dependencies are needed.
- Fixed cfe_remove() return value, as the .remove hook has changed
- Added Krzysztof's Rb.
- Link to v2: https://lore.kernel.org/r/20240620-rp1-cfe-v2-0-b8b48fdba3b3@ideasonboard.com

Changes in v2:
- Change the compatible string back to raspberrypi,rp1-cfe from raspberrypi,rpi5-rp1-cfe
- Drop the references to rp1 headers in the DT binding example. This
  allows compiling the example without the rp1 support.
- Fix missing remap lines for mono formats
- Fix csi2_pad_set_fmt() so that the format can be changed back to the
  sink's format from 16-bit or compressed format.
- Link to v1: https://lore.kernel.org/r/20240318-rp1-cfe-v1-0-ac6d960ff22d@ideasonboard.com

---
Tomi Valkeinen (4):
      media: uapi: Add meta formats for PiSP FE config and stats
      dt-bindings: media: Add bindings for raspberrypi,rp1-cfe
      media: raspberrypi: Add support for RP1-CFE
      media: admin-guide: Document the Raspberry Pi CFE (rp1-cfe)

 .../admin-guide/media/raspberrypi-rp1-cfe.dot      |   27 +
 .../admin-guide/media/raspberrypi-rp1-cfe.rst      |   78 +
 Documentation/admin-guide/media/v4l-drivers.rst    |    1 +
 .../bindings/media/raspberrypi,rp1-cfe.yaml        |   98 +
 .../userspace-api/media/v4l/meta-formats.rst       |    1 +
 .../userspace-api/media/v4l/metafmt-pisp-fe.rst    |   39 +
 MAINTAINERS                                        |    8 +
 drivers/media/platform/raspberrypi/Kconfig         |    1 +
 drivers/media/platform/raspberrypi/Makefile        |    1 +
 drivers/media/platform/raspberrypi/rp1-cfe/Kconfig |   14 +
 .../media/platform/raspberrypi/rp1-cfe/Makefile    |    6 +
 .../media/platform/raspberrypi/rp1-cfe/cfe-fmts.h  |  332 +++
 .../media/platform/raspberrypi/rp1-cfe/cfe-trace.h |  196 ++
 drivers/media/platform/raspberrypi/rp1-cfe/cfe.c   | 2524 ++++++++++++++++++++
 drivers/media/platform/raspberrypi/rp1-cfe/cfe.h   |   43 +
 drivers/media/platform/raspberrypi/rp1-cfe/csi2.c  |  583 +++++
 drivers/media/platform/raspberrypi/rp1-cfe/csi2.h  |   89 +
 drivers/media/platform/raspberrypi/rp1-cfe/dphy.c  |  175 ++
 drivers/media/platform/raspberrypi/rp1-cfe/dphy.h  |   27 +
 .../media/platform/raspberrypi/rp1-cfe/pisp-fe.c   |  581 +++++
 .../media/platform/raspberrypi/rp1-cfe/pisp-fe.h   |   53 +
 drivers/media/v4l2-core/v4l2-ioctl.c               |    2 +
 .../uapi/linux/media/raspberrypi/pisp_fe_config.h  |  273 +++
 .../linux/media/raspberrypi/pisp_fe_statistics.h   |   64 +
 include/uapi/linux/videodev2.h                     |    2 +
 25 files changed, 5218 insertions(+)
---
base-commit: 7c626ce4bae1ac14f60076d00eafe71af30450ba
change-id: 20240314-rp1-cfe-142b628b7214

Best regards,
-- 
Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

[PATCH v3 1/4] media: uapi: Add meta formats for PiSP FE config and stats

[Tomi Valkeinen]

Add two meta formats for PiSP FE: V4L2_META_FMT_RPI_FE_CFG and
V4L2_META_FMT_RPI_FE_STATS. The former is used to provide configuration
for the FE and the latter is used to read the statistics from the FE.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../userspace-api/media/v4l/meta-formats.rst       |  1 +
 .../userspace-api/media/v4l/metafmt-pisp-fe.rst    | 39 ++++++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c               |  2 ++
 include/uapi/linux/videodev2.h                     |  2 ++
 4 files changed, 44 insertions(+)

diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index c6e56b5888bc..86ffb3bc8ade 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -16,6 +16,7 @@ These formats are used for the :ref:`metadata` interface only.
     metafmt-generic
     metafmt-intel-ipu3
     metafmt-pisp-be
+    metafmt-pisp-fe
     metafmt-rkisp1
     metafmt-uvc
     metafmt-vivid
diff --git a/Documentation/userspace-api/media/v4l/metafmt-pisp-fe.rst b/Documentation/userspace-api/media/v4l/metafmt-pisp-fe.rst
new file mode 100644
index 000000000000..fddeada83e4a
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/metafmt-pisp-fe.rst
@@ -0,0 +1,39 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _v4l2-meta-fmt-rpi-fe-cfg:
+
+************************
+V4L2_META_FMT_RPI_FE_CFG
+************************
+
+Raspberry Pi PiSP Front End configuration format
+================================================
+
+The Raspberry Pi PiSP Front End image signal processor is configured by
+userspace by providing a buffer of configuration parameters to the
+`rp1-cfe-fe-config` output video device node using the
+:c:type:`v4l2_meta_format` interface.
+
+The `Raspberry Pi PiSP technical specification
+<https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf>`_
+provide detailed description of the Front End configuration and programming
+model.
+
+.. _v4l2-meta-fmt-rpi-fe-stats:
+
+**************************
+V4L2_META_FMT_RPI_FE_STATS
+**************************
+
+Raspberry Pi PiSP Front End statistics format
+=============================================
+
+The Raspberry Pi PiSP Front End image signal processor provides statistics data
+by writing to a buffer provided via the `rp1-cfe-fe-stats` capture video device
+node using the
+:c:type:`v4l2_meta_format` interface.
+
+The `Raspberry Pi PiSP technical specification
+<https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf>`_
+provide detailed description of the Front End configuration and programming
+model.
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 5eb4d797d259..95c2c4a97966 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1466,6 +1466,8 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 	case V4L2_PIX_FMT_Y212:		descr = "12-bit YUYV Packed"; break;
 	case V4L2_PIX_FMT_Y216:		descr = "16-bit YUYV Packed"; break;
 	case V4L2_META_FMT_RPI_BE_CFG:	descr = "RPi PiSP BE Config format"; break;
+	case V4L2_META_FMT_RPI_FE_CFG:  descr = "RPi PiSP FE Config format"; break;
+	case V4L2_META_FMT_RPI_FE_STATS: descr = "RPi PiSP FE Statistics format"; break;
 	case V4L2_META_FMT_GENERIC_8:	descr = "8-bit Generic Metadata"; break;
 	case V4L2_META_FMT_GENERIC_CSI2_10:	descr = "8-bit Generic Meta, 10b CSI-2"; break;
 	case V4L2_META_FMT_GENERIC_CSI2_12:	descr = "8-bit Generic Meta, 12b CSI-2"; break;
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 4e91362da6da..9f68cd55248f 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -857,6 +857,8 @@ struct v4l2_pix_format {
 
 /* Vendor specific - used for RaspberryPi PiSP */
 #define V4L2_META_FMT_RPI_BE_CFG	v4l2_fourcc('R', 'P', 'B', 'C') /* PiSP BE configuration */
+#define V4L2_META_FMT_RPI_FE_CFG	v4l2_fourcc('R', 'P', 'F', 'C') /* PiSP FE configuration */
+#define V4L2_META_FMT_RPI_FE_STATS	v4l2_fourcc('R', 'P', 'F', 'S') /* PiSP FE stats */
 
 #ifdef __KERNEL__
 /*

-- 
2.43.0

[PATCH v3 2/4] dt-bindings: media: Add bindings for raspberrypi,rp1-cfe

[Tomi Valkeinen]

Add DT bindings for raspberrypi,rp1-cfe.

Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../bindings/media/raspberrypi,rp1-cfe.yaml        | 98 ++++++++++++++++++++++
 1 file changed, 98 insertions(+)

diff --git a/Documentation/devicetree/bindings/media/raspberrypi,rp1-cfe.yaml b/Documentation/devicetree/bindings/media/raspberrypi,rp1-cfe.yaml
new file mode 100644
index 000000000000..851533de2305
--- /dev/null
+++ b/Documentation/devicetree/bindings/media/raspberrypi,rp1-cfe.yaml
@@ -0,0 +1,98 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/media/raspberrypi,rp1-cfe.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Raspberry Pi PiSP Camera Front End
+
+maintainers:
+  - Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
+  - Raspberry Pi Kernel Maintenance <kernel-list@raspberrypi.com>
+
+description: |
+  The Raspberry Pi PiSP Camera Front End is a module in Raspberrypi 5's RP1 I/O
+  controller, that contains:
+  - MIPI D-PHY
+  - MIPI CSI-2 receiver
+  - Simple image processor (called PiSP Front End, or FE)
+
+  The FE documentation is available at:
+  https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf
+
+  The PHY and CSI-2 receiver part have no public documentation.
+
+properties:
+  compatible:
+    items:
+      - const: raspberrypi,rp1-cfe
+
+  reg:
+    items:
+      - description: CSI-2 registers
+      - description: D-PHY registers
+      - description: MIPI CFG (a simple top-level mux) registers
+      - description: FE registers
+
+  interrupts:
+    maxItems: 1
+
+  clocks:
+    maxItems: 1
+
+  port:
+    $ref: /schemas/graph.yaml#/$defs/port-base
+    additionalProperties: false
+    description: CSI-2 RX Port
+
+    properties:
+      endpoint:
+        $ref: video-interfaces.yaml#
+        unevaluatedProperties: false
+
+        properties:
+          data-lanes:
+            minItems: 1
+            maxItems: 4
+
+          clock-lanes:
+            maxItems: 1
+
+        required:
+          - clock-lanes
+          - data-lanes
+
+required:
+  - compatible
+  - reg
+  - interrupts
+  - clocks
+
+additionalProperties: false
+
+examples:
+  - |
+    rp1 {
+      #address-cells = <2>;
+      #size-cells = <2>;
+
+      csi@110000 {
+        compatible = "raspberrypi,rp1-cfe";
+        reg = <0xc0 0x40110000 0x0 0x100>,
+              <0xc0 0x40114000 0x0 0x100>,
+              <0xc0 0x40120000 0x0 0x100>,
+              <0xc0 0x40124000 0x0 0x1000>;
+
+        interrupts = <42>;
+
+        clocks = <&rp1_clocks>;
+
+        port {
+          csi_ep: endpoint {
+            remote-endpoint = <&cam_endpoint>;
+            clock-lanes = <0>;
+            data-lanes = <1 2>;
+          };
+        };
+      };
+    };

-- 
2.43.0

[PATCH v3 4/4] media: admin-guide: Document the Raspberry Pi CFE (rp1-cfe)

[Tomi Valkeinen]

Add documentation for rp1-cfe driver.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../admin-guide/media/raspberrypi-rp1-cfe.dot      | 27 ++++++++
 .../admin-guide/media/raspberrypi-rp1-cfe.rst      | 78 ++++++++++++++++++++++
 Documentation/admin-guide/media/v4l-drivers.rst    |  1 +
 3 files changed, 106 insertions(+)

diff --git a/Documentation/admin-guide/media/raspberrypi-rp1-cfe.dot b/Documentation/admin-guide/media/raspberrypi-rp1-cfe.dot
new file mode 100644
index 000000000000..7717f2291049
--- /dev/null
+++ b/Documentation/admin-guide/media/raspberrypi-rp1-cfe.dot
@@ -0,0 +1,27 @@
+digraph board {
+	rankdir=TB
+	n00000001 [label="{{<port0> 0} | csi2\n/dev/v4l-subdev0 | {<port1> 1 | <port2> 2 | <port3> 3 | <port4> 4}}", shape=Mrecord, style=filled, fillcolor=green]
+	n00000001:port1 -> n00000011 [style=dashed]
+	n00000001:port1 -> n00000007:port0
+	n00000001:port2 -> n00000015
+	n00000001:port2 -> n00000007:port0 [style=dashed]
+	n00000001:port3 -> n00000019 [style=dashed]
+	n00000001:port3 -> n00000007:port0 [style=dashed]
+	n00000001:port4 -> n0000001d [style=dashed]
+	n00000001:port4 -> n00000007:port0 [style=dashed]
+	n00000007 [label="{{<port0> 0 | <port1> 1} | pisp-fe\n/dev/v4l-subdev1 | {<port2> 2 | <port3> 3 | <port4> 4}}", shape=Mrecord, style=filled, fillcolor=green]
+	n00000007:port2 -> n00000021
+	n00000007:port3 -> n00000025 [style=dashed]
+	n00000007:port4 -> n00000029
+	n0000000d [label="{imx219 6-0010\n/dev/v4l-subdev2 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green]
+	n0000000d:port0 -> n00000001:port0 [style=bold]
+	n00000011 [label="rp1-cfe-csi2-ch0\n/dev/video0", shape=box, style=filled, fillcolor=yellow]
+	n00000015 [label="rp1-cfe-csi2-ch1\n/dev/video1", shape=box, style=filled, fillcolor=yellow]
+	n00000019 [label="rp1-cfe-csi2-ch2\n/dev/video2", shape=box, style=filled, fillcolor=yellow]
+	n0000001d [label="rp1-cfe-csi2-ch3\n/dev/video3", shape=box, style=filled, fillcolor=yellow]
+	n00000021 [label="rp1-cfe-fe-image0\n/dev/video4", shape=box, style=filled, fillcolor=yellow]
+	n00000025 [label="rp1-cfe-fe-image1\n/dev/video5", shape=box, style=filled, fillcolor=yellow]
+	n00000029 [label="rp1-cfe-fe-stats\n/dev/video6", shape=box, style=filled, fillcolor=yellow]
+	n0000002d [label="rp1-cfe-fe-config\n/dev/video7", shape=box, style=filled, fillcolor=yellow]
+	n0000002d -> n00000007:port1
+}
diff --git a/Documentation/admin-guide/media/raspberrypi-rp1-cfe.rst b/Documentation/admin-guide/media/raspberrypi-rp1-cfe.rst
new file mode 100644
index 000000000000..668d978a9875
--- /dev/null
+++ b/Documentation/admin-guide/media/raspberrypi-rp1-cfe.rst
@@ -0,0 +1,78 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+============================================
+Raspberry Pi PiSP Camera Front End (rp1-cfe)
+============================================
+
+The PiSP Camera Front End
+=========================
+
+The PiSP Camera Front End (CFE) is a module which combines a CSI-2 receiver with
+a simple ISP, called the Front End (FE).
+
+The CFE has four DMA engines and can write frames from four separate streams
+received from the CSI-2 to the memory. One of those streams can also be routed
+directly to the FE, which can do minimal image processing, write two versions
+(e.g. non-scaled and downscaled versions) of the received frames to memory and
+provide statistics of the received frames.
+
+The FE registers are documented in the `Raspberry Pi Image Signal Processor
+(ISP) Specification document
+<https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf>`_,
+and example code for FE can be found in `libpisp
+<https://github.com/raspberrypi/libpisp>`_.
+
+The rp1-cfe driver
+==================
+
+The Raspberry Pi PiSP Camera Front End (rp1-cfe) driver is located under
+drivers/media/platform/raspberrypi/rp1-cfe. It uses the `V4L2 API` to register
+a number of video capture and output devices, the `V4L2 subdev API` to register
+subdevices for the CSI-2 received and the FE that connects the video devices in
+a single media graph realized using the `Media Controller (MC) API`.
+
+The media topology registered by the `rp1-cfe` driver, in this particular
+example connected to an imx219 sensor, is the following one:
+
+.. _rp1-cfe-topology:
+
+.. kernel-figure:: raspberrypi-rp1-cfe.dot
+    :alt:   Diagram of an example media pipeline topology
+    :align: center
+
+The media graph contains the following video device nodes:
+
+- rp1-cfe-csi2-ch0: capture device for the first CSI-2 stream
+- rp1-cfe-csi2-ch1: capture device for the second CSI-2 stream
+- rp1-cfe-csi2-ch2: capture device for the third CSI-2 stream
+- rp1-cfe-csi2-ch3: capture device for the fourth CSI-2 stream
+- rp1-cfe-fe-image0: capture device for the first FE output
+- rp1-cfe-fe-image1: capture device for the second FE output
+- rp1-cfe-fe-stats: capture device for the FE statistics
+- rp1-cfe-fe-config: output device for FE configuration
+
+rp1-cfe-csi2-chX
+----------------
+
+The rp1-cfe-csi2-chX capture devices are normal V4L2 capture devices which
+can be used to capture video frames or metadata received from the CSI-2.
+
+rp1-cfe-fe-image0, rp1-cfe-fe-image1
+------------------------------------
+
+The rp1-cfe-fe-image0 and rp1-cfe-fe-image1 capture devices are used to write
+the processed frames to memory.
+
+rp1-cfe-fe-stats
+----------------
+
+The format of the FE statistics buffer is defined by
+:c:type:`pisp_statistics` C structure and the meaning of each parameter is
+described in the `PiSP specification` document.
+
+rp1-cfe-fe-config
+-----------------
+
+The format of the FE configuration buffer is defined by
+:c:type:`pisp_fe_config` C structure and the meaning of each parameter is
+described in the `PiSP specification` document.
diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst
index b6af448b9fe9..61da154e079a 100644
--- a/Documentation/admin-guide/media/v4l-drivers.rst
+++ b/Documentation/admin-guide/media/v4l-drivers.rst
@@ -26,6 +26,7 @@ Video4Linux (V4L) driver-specific documentation
 	raspberrypi-pisp-be
 	rcar-fdp1
 	rkisp1
+	raspberrypi-rp1-cfe
 	saa7134
 	si470x
 	si4713

-- 
2.43.0

Re: [PATCH v3 3/4] media: raspberrypi: Add support for RP1-CFE

[Sakari Ailus]

Moi,

On Mon, Sep 02, 2024 at 01:05:42PM +0300, Tomi Valkeinen wrote:
> Hi Sakari,
> 
> Thanks for the review!

You're welcome.

> > > +#define cfe_dbg(fmt, arg...) dev_dbg(&cfe->pdev->dev, fmt, ##arg)
> > 
> > cfe should be an argument to cfe_dbg().
> 
> Why? This, and the ones below, is an internal macro to make it easier and
> shorter to do prints. Adding the parameter gives no benefit that I can see.

Generally macros shouldn't expect certain variables not defined on the same
level the macros themselves. It gets harder to maintain this way.

> > > +#define node_supports_image_output(node) \
> > > +	(!!(node_desc[(node)->id].caps & V4L2_CAP_VIDEO_CAPTURE))
> > 
> > No need to cast to bool through !!. Same below.
> 
> I like my bools to be bools, not ints... But at the same time, I don't see
> how that would cause issues in the uses we have in this driver. So I'll drop
> these.

Alternatively, explicitly cast to bool. But I don't think it's needed.

> 
> > > +#define node_supports_meta_output(node) \
> > > +	(!!(node_desc[(node)->id].caps & V4L2_CAP_META_CAPTURE))
> > > +#define node_supports_image_input(node) \
> > > +	(!!(node_desc[(node)->id].caps & V4L2_CAP_VIDEO_OUTPUT))
> > > +#define node_supports_meta_input(node) \
> > > +	(!!(node_desc[(node)->id].caps & V4L2_CAP_META_OUTPUT))
> > > +#define node_supports_image(node) \
> > > +	(node_supports_image_output(node) || node_supports_image_input(node))
> > > +#define node_supports_meta(node) \
> > > +	(node_supports_meta_output(node) || node_supports_meta_input(node))
> > > +
> > > +#define is_image_output_node(node) \
> > > +	((node)->buffer_queue.type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> > > +#define is_image_input_node(node) \
> > > +	((node)->buffer_queue.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
> > > +#define is_image_node(node) \
> > > +	(is_image_output_node(node) || is_image_input_node(node))
> > > +#define is_meta_output_node(node) \
> > > +	((node)->buffer_queue.type == V4L2_BUF_TYPE_META_CAPTURE)
> > > +#define is_meta_input_node(node) \
> > > +	((node)->buffer_queue.type == V4L2_BUF_TYPE_META_OUTPUT)
> > > +#define is_meta_node(node) \
> > > +	(is_meta_output_node(node) || is_meta_input_node(node))
> > > +
> > > +/* To track state across all nodes. */
> > > +#define NUM_STATES		5

This might be nicer if declared as last.

> > > +#define NODE_REGISTERED		BIT(0)
> > > +#define NODE_ENABLED		BIT(1)
> > > +#define NODE_STREAMING		BIT(2)
> > > +#define FS_INT			BIT(3)
> > > +#define FE_INT			BIT(4)

...

> > > +static int cfe_start_channel(struct cfe_node *node)
> > > +{
> > > +	struct cfe_device *cfe = node->cfe;
> > > +	struct v4l2_subdev_state *state;
> > > +	struct v4l2_mbus_framefmt *source_fmt;
> > > +	const struct cfe_fmt *fmt;
> > > +	unsigned long flags;
> > > +	bool start_fe;
> > > +	int ret;
> > > +
> > > +	cfe_dbg("%s: [%s]\n", __func__, node_desc[node->id].name);
> > 
> > This looks like a development time leftover. There are quite a few such
> > prints that provide little information anyway. How about removing them all?
> 
> These are very valuable when testing, fixing or improving the driver. If I
> were to remove them, I would just have to add them back whenever I'd be
> doing something with the driver and things would not work perfectly.
> 
> The debug prints we have are all low frequency. There's a bunch printed when
> starting the streaming and when stopping it, but the debug prints are not
> used while streaming is on-going, and instead we have trace events for that.

I'm fine with debug prints when they do print useful information but you
have many of them just in the beginning of the function, printing only the
function name (and possibly the node name). I'd remove those, except in
cases where calling the function itself is useful information, such as on
returning the buffers.

> > > +static int cfe_link_node_pads(struct cfe_device *cfe)
> > > +{
> > > +	int ret;
> > > +	int pad;
> > > +
> > > +	/* Source -> CSI2 */
> > > +
> > > +	pad = media_entity_get_fwnode_pad(&cfe->source_sd->entity,
> > > +					  cfe->remote_ep_fwnode,
> > > +					  MEDIA_PAD_FL_SOURCE);
> > > +	if (pad < 0) {
> > > +		cfe_err("Source %s has no connected source pad\n",
> > > +			cfe->source_sd->name);
> > > +		return pad;
> > > +	}
> > > +
> > > +	cfe->source_pad = pad;
> > > +
> > > +	ret = media_create_pad_link(&cfe->source_sd->entity, pad,
> > > +				    &cfe->csi2.sd.entity, CSI2_PAD_SINK,
> > > +				    MEDIA_LNK_FL_IMMUTABLE |
> > > +				    MEDIA_LNK_FL_ENABLED);
> > > +	if (ret)
> > > +		return ret;
> > > +
> > > +	for (unsigned int i = 0; i < CSI2_NUM_CHANNELS; i++) {
> > > +		struct cfe_node *node = &cfe->node[i];
> > > +
> > > +		if (!check_state(cfe, NODE_REGISTERED, i))
> > > +			continue;
> > > +
> > > +		/* CSI2 channel # -> /dev/video# */
> > > +		ret = media_create_pad_link(&cfe->csi2.sd.entity,
> > > +					    node_desc[i].link_pad,
> > > +					    &node->video_dev.entity, 0, 0);
> > > +		if (ret)
> > > +			return ret;
> > > +
> > > +		if (node_supports_image(node)) {
> > > +			/* CSI2 channel # -> FE Input */
> > > +			ret = media_create_pad_link(&cfe->csi2.sd.entity,
> > > +						    node_desc[i].link_pad,
> > > +						    &cfe->fe.sd.entity,
> > > +						    FE_STREAM_PAD, 0);
> > > +			if (ret)
> > > +				return ret;
> > > +		}
> > > +	}
> > > +
> > > +	for (unsigned int i = CSI2_NUM_CHANNELS; i < NUM_NODES; i++) {
> > > +		struct cfe_node *node = &cfe->node[i];
> > > +		struct media_entity *src, *dst;
> > > +		unsigned int src_pad, dst_pad;
> > > +
> > > +		if (node_desc[i].pad_flags & MEDIA_PAD_FL_SINK) {
> > > +			/* FE -> /dev/video# */
> > > +			src = &cfe->fe.sd.entity;
> > > +			src_pad = node_desc[i].link_pad;
> > > +			dst = &node->video_dev.entity;
> > > +			dst_pad = 0;
> > > +		} else {
> > > +			/* /dev/video# -> FE */
> > > +			dst = &cfe->fe.sd.entity;
> > > +			dst_pad = node_desc[i].link_pad;
> > > +			src = &node->video_dev.entity;
> > > +			src_pad = 0;
> > > +		}
> > > +
> > > +		ret = media_create_pad_link(src, src_pad, dst, dst_pad, 0);
> > > +		if (ret)
> > > +			return ret;
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static int cfe_probe_complete(struct cfe_device *cfe)
> > > +{
> > > +	int ret;
> > > +
> > > +	cfe->v4l2_dev.notify = cfe_notify;
> > > +
> > > +	for (unsigned int i = 0; i < NUM_NODES; i++) {
> > > +		ret = cfe_register_node(cfe, i);
> > > +		if (ret) {
> > > +			cfe_err("Unable to register video node %u.\n", i);
> > > +			goto unregister;
> > > +		}
> > > +	}
> > > +
> > > +	ret = cfe_link_node_pads(cfe);
> > > +	if (ret) {
> > > +		cfe_err("Unable to link node pads.\n");
> > > +		goto unregister;
> > > +	}
> > > +
> > > +	ret = v4l2_device_register_subdev_nodes(&cfe->v4l2_dev);
> > > +	if (ret) {
> > > +		cfe_err("Unable to register subdev nodes.\n");
> > > +		goto unregister;
> > > +	}
> > > +
> > > +	return 0;
> > > +
> > > +unregister:
> > > +	cfe_unregister_nodes(cfe);
> > > +	return ret;
> > > +}
> > > +
> > > +static int cfe_async_bound(struct v4l2_async_notifier *notifier,
> > > +			   struct v4l2_subdev *subdev,
> > > +			   struct v4l2_async_connection *asd)
> > > +{
> > > +	struct cfe_device *cfe = to_cfe_device(notifier->v4l2_dev);
> > > +
> > > +	if (cfe->source_sd) {
> > > +		cfe_err("Rejecting subdev %s (Already set!!)", subdev->name);
> > > +		return 0;
> > > +	}
> > > +
> > > +	cfe->source_sd = subdev;
> > > +
> > > +	cfe_dbg("Using source %s for capture\n", subdev->name);
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +static int cfe_async_complete(struct v4l2_async_notifier *notifier)
> > > +{
> > > +	struct cfe_device *cfe = to_cfe_device(notifier->v4l2_dev);
> > > +
> > > +	return cfe_probe_complete(cfe);
> > > +}
> > > +
> > > +static const struct v4l2_async_notifier_operations cfe_async_ops = {
> > > +	.bound = cfe_async_bound,
> > > +	.complete = cfe_async_complete,
> > > +};
> > > +
> > > +static int cfe_register_async_nf(struct cfe_device *cfe)
> > > +{
> > > +	struct platform_device *pdev = cfe->pdev;
> > > +	struct v4l2_fwnode_endpoint ep = { .bus_type = V4L2_MBUS_CSI2_DPHY };
> > > +	int ret = -EINVAL;

Is the assignment necessary?

> > > +	struct fwnode_handle *local_ep_fwnode;
> > > +	struct fwnode_handle *remote_ep_fwnode;
> > > +	struct v4l2_async_connection *asd;
> > > +
> > > +	local_ep_fwnode = fwnode_graph_get_endpoint_by_id(pdev->dev.fwnode, 0, 0, 0);
> > > +	if (!local_ep_fwnode) {
> > > +		cfe_err("Failed to find local endpoint fwnode\n");
> > > +		return -ENODEV;
> > > +	}
> > > +
> > > +	remote_ep_fwnode = fwnode_graph_get_remote_endpoint(local_ep_fwnode);
> > > +	if (!remote_ep_fwnode) {
> > > +		cfe_err("Failed to find remote endpoint fwnode\n");
> > > +		ret = -ENODEV;
> > > +		goto err_put_local_fwnode;
> > > +	}
> > > +
> > > +	/* Parse the local endpoint and validate its configuration. */
> > > +	v4l2_fwnode_endpoint_parse(local_ep_fwnode, &ep);

You'll need to check the return value here.

> > > +
> > > +	if (ep.bus_type != V4L2_MBUS_CSI2_DPHY) {

This check is redundant.

> > > +		cfe_err("endpoint node type != CSI2\n");
> > > +		ret = -EINVAL;
> > > +		goto err_put_remote_fwnode;
> > > +	}
> > > +
> > > +	for (unsigned int lane = 0; lane < ep.bus.mipi_csi2.num_data_lanes; lane++) {
> > > +		if (ep.bus.mipi_csi2.data_lanes[lane] != lane + 1) {
> > > +			cfe_err("subdevice %pfwf: data lanes reordering not supported\n",
> > > +				remote_ep_fwnode);
> > > +			ret = -EINVAL;
> > > +			goto err_put_remote_fwnode;
> > > +		}
> > > +	}
> > > +
> > > +	cfe->csi2.dphy.max_lanes = ep.bus.mipi_csi2.num_data_lanes;
> > > +	cfe->csi2.bus_flags = ep.bus.mipi_csi2.flags;
> > > +
> > > +	cfe->remote_ep_fwnode = remote_ep_fwnode;
> > > +
> > > +	cfe_dbg("source %pfwf: %u data lanes, flags=0x%08x\n",
> > > +		remote_ep_fwnode, cfe->csi2.dphy.max_lanes, cfe->csi2.bus_flags);
> > > +
> > > +	/* Initialize and register the async notifier. */
> > > +	v4l2_async_nf_init(&cfe->notifier, &cfe->v4l2_dev);
> > > +	cfe->notifier.ops = &cfe_async_ops;
> > > +
> > > +	asd = v4l2_async_nf_add_fwnode(&cfe->notifier, remote_ep_fwnode,
> > > +				       struct v4l2_async_connection);
> > 
> > Could you use v4l2_async_nf_add_fwnode_remote() and just not bother with
> > remote_ep_fwnode at all?
> 
> I need the remote_ep_fwnode in cfe_link_node_pads() when creating the links.
> 
> Is there some other way to get the remote pad so that I can call the
> media_create_pad_link()?

Could you use v4l2_create_fwnode_links_to_pad() for that?

> 
> > > +	if (IS_ERR(asd)) {
> > > +		ret = PTR_ERR(asd);
> > > +		cfe_err("Error adding subdevice: %d\n", ret);
> > > +		goto err_put_remote_fwnode;
> > > +	}
> > > +
> > > +	ret = v4l2_async_nf_register(&cfe->notifier);
> > > +	if (ret) {
> > > +		cfe_err("Error registering async notifier: %d\n", ret);
> > > +		goto err_nf_cleanup;
> > > +	}
> > > +
> > > +	fwnode_handle_put(local_ep_fwnode);
> > > +
> > > +	return 0;
> > > +
> > > +err_nf_cleanup:
> > > +	v4l2_async_nf_cleanup(&cfe->notifier);
> > > +err_put_remote_fwnode:
> > > +	fwnode_handle_put(remote_ep_fwnode);
> > > +err_put_local_fwnode:
> > > +	fwnode_handle_put(local_ep_fwnode);
> > > +
> > > +	return ret;
> > > +}

...

> > > +static void cfe_remove(struct platform_device *pdev)
> > > +{
> > > +	struct cfe_device *cfe = platform_get_drvdata(pdev);
> > > +
> > > +	debugfs_remove(cfe->debugfs);
> > > +
> > > +	v4l2_async_nf_unregister(&cfe->notifier);
> > > +	v4l2_async_nf_cleanup(&cfe->notifier);
> > > +
> > > +	media_device_unregister(&cfe->mdev);
> > 
> > Do you think you might have the time to write patch that would convert the
> > driver to use media device refcounting? It's not needed for upstreaming
> > though, but would be nice to have such a new driver to be converted when we
> > (hopefully not in too distant future) could merge that set.
> 
> I had a quick try. I noticed that the current driver will crash if streaming
> is active when unbinding/unloading, so that needs fixing anyway.
> 
> But with the v4 series ([PATCH v4 00/26] Media device lifetime management),
> I hit WARN_ON(!vdev->release) in __video_register_device(). I haven't
> studied the series that much, but I understood vdev's release will be
> handled via the media device. Is that not right?

Let's discuss this online.

> > > +	debugfs_create_file("csi2_regs", 0444, debugfs, csi2, &csi2_regs_fops);
> > 
> > Should this reflect the device name? This will currently only work with one
> > such device in the system.
> 
> These files are in a device specific directory, e.g.
> "rp1-cfe:1f00110000.csi/".

Ack.

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v3 3/4] media: raspberrypi: Add support for RP1-CFE

[Tomi Valkeinen]

On 04/09/2024 10:07, Sakari Ailus wrote:
> Moi,
> 
> On Mon, Sep 02, 2024 at 01:05:42PM +0300, Tomi Valkeinen wrote:
>> Hi Sakari,
>>
>> Thanks for the review!
> 
> You're welcome.
> 
>>>> +#define cfe_dbg(fmt, arg...) dev_dbg(&cfe->pdev->dev, fmt, ##arg)
>>>
>>> cfe should be an argument to cfe_dbg().
>>
>> Why? This, and the ones below, is an internal macro to make it easier and
>> shorter to do prints. Adding the parameter gives no benefit that I can see.
> 
> Generally macros shouldn't expect certain variables not defined on the same
> level the macros themselves. It gets harder to maintain this way.

I agree, but I'm fine taking shortcuts in private macros to make the 
code a bit shorter and more readable.

In any case, I don't feel strongly about this so I'll make the change, 
it's an easy find and replace.

>>>> +#define node_supports_image_output(node) \
>>>> +	(!!(node_desc[(node)->id].caps & V4L2_CAP_VIDEO_CAPTURE))
>>>
>>> No need to cast to bool through !!. Same below.
>>
>> I like my bools to be bools, not ints... But at the same time, I don't see
>> how that would cause issues in the uses we have in this driver. So I'll drop
>> these.
> 
> Alternatively, explicitly cast to bool. But I don't think it's needed.
> 
>>
>>>> +#define node_supports_meta_output(node) \
>>>> +	(!!(node_desc[(node)->id].caps & V4L2_CAP_META_CAPTURE))
>>>> +#define node_supports_image_input(node) \
>>>> +	(!!(node_desc[(node)->id].caps & V4L2_CAP_VIDEO_OUTPUT))
>>>> +#define node_supports_meta_input(node) \
>>>> +	(!!(node_desc[(node)->id].caps & V4L2_CAP_META_OUTPUT))
>>>> +#define node_supports_image(node) \
>>>> +	(node_supports_image_output(node) || node_supports_image_input(node))
>>>> +#define node_supports_meta(node) \
>>>> +	(node_supports_meta_output(node) || node_supports_meta_input(node))
>>>> +
>>>> +#define is_image_output_node(node) \
>>>> +	((node)->buffer_queue.type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
>>>> +#define is_image_input_node(node) \
>>>> +	((node)->buffer_queue.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
>>>> +#define is_image_node(node) \
>>>> +	(is_image_output_node(node) || is_image_input_node(node))
>>>> +#define is_meta_output_node(node) \
>>>> +	((node)->buffer_queue.type == V4L2_BUF_TYPE_META_CAPTURE)
>>>> +#define is_meta_input_node(node) \
>>>> +	((node)->buffer_queue.type == V4L2_BUF_TYPE_META_OUTPUT)
>>>> +#define is_meta_node(node) \
>>>> +	(is_meta_output_node(node) || is_meta_input_node(node))
>>>> +
>>>> +/* To track state across all nodes. */
>>>> +#define NUM_STATES		5
> 
> This might be nicer if declared as last.

Sure.

>>>> +#define NODE_REGISTERED		BIT(0)
>>>> +#define NODE_ENABLED		BIT(1)
>>>> +#define NODE_STREAMING		BIT(2)
>>>> +#define FS_INT			BIT(3)
>>>> +#define FE_INT			BIT(4)
> 
> ...
> 
>>>> +static int cfe_start_channel(struct cfe_node *node)
>>>> +{
>>>> +	struct cfe_device *cfe = node->cfe;
>>>> +	struct v4l2_subdev_state *state;
>>>> +	struct v4l2_mbus_framefmt *source_fmt;
>>>> +	const struct cfe_fmt *fmt;
>>>> +	unsigned long flags;
>>>> +	bool start_fe;
>>>> +	int ret;
>>>> +
>>>> +	cfe_dbg("%s: [%s]\n", __func__, node_desc[node->id].name);
>>>
>>> This looks like a development time leftover. There are quite a few such
>>> prints that provide little information anyway. How about removing them all?
>>
>> These are very valuable when testing, fixing or improving the driver. If I
>> were to remove them, I would just have to add them back whenever I'd be
>> doing something with the driver and things would not work perfectly.
>>
>> The debug prints we have are all low frequency. There's a bunch printed when
>> starting the streaming and when stopping it, but the debug prints are not
>> used while streaming is on-going, and instead we have trace events for that.
> 
> I'm fine with debug prints when they do print useful information but you
> have many of them just in the beginning of the function, printing only the
> function name (and possibly the node name). I'd remove those, except in
> cases where calling the function itself is useful information, such as on
> returning the buffers.

I have removed a few debug prints, but I ended up adding a few too.

The useful information here in many cases is the function name and the 
node name. With multiple streams, and with the different possibilities 
in handling enable and disable (start streaming when the first video 
node is enabled? or only when all of them are enabled? what happens when 
one of the enabled nodes is disabled? is the csi side enabled along with 
the fe, and which csi channel?), it's very beneficial to see what goes on.

>>>> +static int cfe_link_node_pads(struct cfe_device *cfe)
>>>> +{
>>>> +	int ret;
>>>> +	int pad;
>>>> +
>>>> +	/* Source -> CSI2 */
>>>> +
>>>> +	pad = media_entity_get_fwnode_pad(&cfe->source_sd->entity,
>>>> +					  cfe->remote_ep_fwnode,
>>>> +					  MEDIA_PAD_FL_SOURCE);
>>>> +	if (pad < 0) {
>>>> +		cfe_err("Source %s has no connected source pad\n",
>>>> +			cfe->source_sd->name);
>>>> +		return pad;
>>>> +	}
>>>> +
>>>> +	cfe->source_pad = pad;
>>>> +
>>>> +	ret = media_create_pad_link(&cfe->source_sd->entity, pad,
>>>> +				    &cfe->csi2.sd.entity, CSI2_PAD_SINK,
>>>> +				    MEDIA_LNK_FL_IMMUTABLE |
>>>> +				    MEDIA_LNK_FL_ENABLED);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	for (unsigned int i = 0; i < CSI2_NUM_CHANNELS; i++) {
>>>> +		struct cfe_node *node = &cfe->node[i];
>>>> +
>>>> +		if (!check_state(cfe, NODE_REGISTERED, i))
>>>> +			continue;
>>>> +
>>>> +		/* CSI2 channel # -> /dev/video# */
>>>> +		ret = media_create_pad_link(&cfe->csi2.sd.entity,
>>>> +					    node_desc[i].link_pad,
>>>> +					    &node->video_dev.entity, 0, 0);
>>>> +		if (ret)
>>>> +			return ret;
>>>> +
>>>> +		if (node_supports_image(node)) {
>>>> +			/* CSI2 channel # -> FE Input */
>>>> +			ret = media_create_pad_link(&cfe->csi2.sd.entity,
>>>> +						    node_desc[i].link_pad,
>>>> +						    &cfe->fe.sd.entity,
>>>> +						    FE_STREAM_PAD, 0);
>>>> +			if (ret)
>>>> +				return ret;
>>>> +		}
>>>> +	}
>>>> +
>>>> +	for (unsigned int i = CSI2_NUM_CHANNELS; i < NUM_NODES; i++) {
>>>> +		struct cfe_node *node = &cfe->node[i];
>>>> +		struct media_entity *src, *dst;
>>>> +		unsigned int src_pad, dst_pad;
>>>> +
>>>> +		if (node_desc[i].pad_flags & MEDIA_PAD_FL_SINK) {
>>>> +			/* FE -> /dev/video# */
>>>> +			src = &cfe->fe.sd.entity;
>>>> +			src_pad = node_desc[i].link_pad;
>>>> +			dst = &node->video_dev.entity;
>>>> +			dst_pad = 0;
>>>> +		} else {
>>>> +			/* /dev/video# -> FE */
>>>> +			dst = &cfe->fe.sd.entity;
>>>> +			dst_pad = node_desc[i].link_pad;
>>>> +			src = &node->video_dev.entity;
>>>> +			src_pad = 0;
>>>> +		}
>>>> +
>>>> +		ret = media_create_pad_link(src, src_pad, dst, dst_pad, 0);
>>>> +		if (ret)
>>>> +			return ret;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int cfe_probe_complete(struct cfe_device *cfe)
>>>> +{
>>>> +	int ret;
>>>> +
>>>> +	cfe->v4l2_dev.notify = cfe_notify;
>>>> +
>>>> +	for (unsigned int i = 0; i < NUM_NODES; i++) {
>>>> +		ret = cfe_register_node(cfe, i);
>>>> +		if (ret) {
>>>> +			cfe_err("Unable to register video node %u.\n", i);
>>>> +			goto unregister;
>>>> +		}
>>>> +	}
>>>> +
>>>> +	ret = cfe_link_node_pads(cfe);
>>>> +	if (ret) {
>>>> +		cfe_err("Unable to link node pads.\n");
>>>> +		goto unregister;
>>>> +	}
>>>> +
>>>> +	ret = v4l2_device_register_subdev_nodes(&cfe->v4l2_dev);
>>>> +	if (ret) {
>>>> +		cfe_err("Unable to register subdev nodes.\n");
>>>> +		goto unregister;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +
>>>> +unregister:
>>>> +	cfe_unregister_nodes(cfe);
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static int cfe_async_bound(struct v4l2_async_notifier *notifier,
>>>> +			   struct v4l2_subdev *subdev,
>>>> +			   struct v4l2_async_connection *asd)
>>>> +{
>>>> +	struct cfe_device *cfe = to_cfe_device(notifier->v4l2_dev);
>>>> +
>>>> +	if (cfe->source_sd) {
>>>> +		cfe_err("Rejecting subdev %s (Already set!!)", subdev->name);
>>>> +		return 0;
>>>> +	}
>>>> +
>>>> +	cfe->source_sd = subdev;
>>>> +
>>>> +	cfe_dbg("Using source %s for capture\n", subdev->name);
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int cfe_async_complete(struct v4l2_async_notifier *notifier)
>>>> +{
>>>> +	struct cfe_device *cfe = to_cfe_device(notifier->v4l2_dev);
>>>> +
>>>> +	return cfe_probe_complete(cfe);
>>>> +}
>>>> +
>>>> +static const struct v4l2_async_notifier_operations cfe_async_ops = {
>>>> +	.bound = cfe_async_bound,
>>>> +	.complete = cfe_async_complete,
>>>> +};
>>>> +
>>>> +static int cfe_register_async_nf(struct cfe_device *cfe)
>>>> +{
>>>> +	struct platform_device *pdev = cfe->pdev;
>>>> +	struct v4l2_fwnode_endpoint ep = { .bus_type = V4L2_MBUS_CSI2_DPHY };
>>>> +	int ret = -EINVAL;
> 
> Is the assignment necessary?

No, doesn't look like it.

>>>> +	struct fwnode_handle *local_ep_fwnode;
>>>> +	struct fwnode_handle *remote_ep_fwnode;
>>>> +	struct v4l2_async_connection *asd;
>>>> +
>>>> +	local_ep_fwnode = fwnode_graph_get_endpoint_by_id(pdev->dev.fwnode, 0, 0, 0);
>>>> +	if (!local_ep_fwnode) {
>>>> +		cfe_err("Failed to find local endpoint fwnode\n");
>>>> +		return -ENODEV;
>>>> +	}
>>>> +
>>>> +	remote_ep_fwnode = fwnode_graph_get_remote_endpoint(local_ep_fwnode);
>>>> +	if (!remote_ep_fwnode) {
>>>> +		cfe_err("Failed to find remote endpoint fwnode\n");
>>>> +		ret = -ENODEV;
>>>> +		goto err_put_local_fwnode;
>>>> +	}
>>>> +
>>>> +	/* Parse the local endpoint and validate its configuration. */
>>>> +	v4l2_fwnode_endpoint_parse(local_ep_fwnode, &ep);
> 
> You'll need to check the return value here.

I'll add that.

>>>> +
>>>> +	if (ep.bus_type != V4L2_MBUS_CSI2_DPHY) {
> 
> This check is redundant.

Indeed.

>>>> +		cfe_err("endpoint node type != CSI2\n");
>>>> +		ret = -EINVAL;
>>>> +		goto err_put_remote_fwnode;
>>>> +	}
>>>> +
>>>> +	for (unsigned int lane = 0; lane < ep.bus.mipi_csi2.num_data_lanes; lane++) {
>>>> +		if (ep.bus.mipi_csi2.data_lanes[lane] != lane + 1) {
>>>> +			cfe_err("subdevice %pfwf: data lanes reordering not supported\n",
>>>> +				remote_ep_fwnode);
>>>> +			ret = -EINVAL;
>>>> +			goto err_put_remote_fwnode;
>>>> +		}
>>>> +	}
>>>> +
>>>> +	cfe->csi2.dphy.max_lanes = ep.bus.mipi_csi2.num_data_lanes;
>>>> +	cfe->csi2.bus_flags = ep.bus.mipi_csi2.flags;
>>>> +
>>>> +	cfe->remote_ep_fwnode = remote_ep_fwnode;
>>>> +
>>>> +	cfe_dbg("source %pfwf: %u data lanes, flags=0x%08x\n",
>>>> +		remote_ep_fwnode, cfe->csi2.dphy.max_lanes, cfe->csi2.bus_flags);
>>>> +
>>>> +	/* Initialize and register the async notifier. */
>>>> +	v4l2_async_nf_init(&cfe->notifier, &cfe->v4l2_dev);
>>>> +	cfe->notifier.ops = &cfe_async_ops;
>>>> +
>>>> +	asd = v4l2_async_nf_add_fwnode(&cfe->notifier, remote_ep_fwnode,
>>>> +				       struct v4l2_async_connection);
>>>
>>> Could you use v4l2_async_nf_add_fwnode_remote() and just not bother with
>>> remote_ep_fwnode at all?
>>
>> I need the remote_ep_fwnode in cfe_link_node_pads() when creating the links.
>>
>> Is there some other way to get the remote pad so that I can call the
>> media_create_pad_link()?
> 
> Could you use v4l2_create_fwnode_links_to_pad() for that?

Yes, seems to work. However, I don't know why it works, as the docs say 
the sink has to implement .get_fwnode_pad(), which I don't have, yet 
everything seems to work fine... Ah, the code looks for the first sink 
pad if .get_fwnode_pad() is not implemented. So either the code or the 
doc is not right.

I still need to get the remote pad number, which I previously got from 
media_entity_get_fwnode_pad(), but I can now create the link first with 
v4l2_create_fwnode_links_to_pad() and then get the pad with 
media_pad_remote_pad_unique().

  Tomi