Re: [PATCH v14 00/18] media: rockchip: add a driver for the rockchip camera interface

[Michael Riesch <michael.riesch@collabora.com>]

Hi Laurent,

On 11/11/25 01:06, Laurent Pinchart wrote:
> On Mon, Nov 10, 2025 at 11:29:57AM +0100, Michael Riesch wrote:
>> On 11/10/25 09:43, Sakari Ailus wrote:
>>> On Fri, Nov 07, 2025 at 09:51:37PM +0100, Michael Riesch wrote:
>>>> On 11/7/25 19:54, Laurent Pinchart wrote:
>>>>> On Fri, Nov 07, 2025 at 07:41:59PM +0100, Michael Riesch wrote:
>>>>>> On 11/7/25 18:32, Sakari Ailus wrote:
>>>>>>> On Fri, Oct 24, 2025 at 02:51:29PM +0200, Michael Riesch via B4 Relay wrote:
>>>>>>>> Habidere,
>>>>>>>>
>>>>>>>> This series introduces support for the Rockchip Camera Interface (CIF),
>>>>>>>> which is featured in many Rockchip SoCs in different variations.
>>>>>>>> For example, the PX30 Video Input Processor (VIP) is able to receive
>>>>>>>> video data via the Digital Video Port (DVP, a parallel data interface)
>>>>>>>> and transfer it into system memory using a double-buffering mechanism
>>>>>>>> called ping-pong mode.
>>>>>>>> The RK3568 Video Capture (VICAP) unit, on the other hand, features a
>>>>>>>> DVP and a MIPI CSI-2 receiver that can receive video data independently
>>>>>>>> (both using the ping-pong scheme).
>>>>>>>> The different variants may have additional features, such as scaling
>>>>>>>> and/or cropping.
>>>>>>>> Finally, the RK3588 VICAP unit constitutes an essential piece of the
>>>>>>>> camera interface with one DVP, six MIPI CSI-2 receivers, scale/crop
>>>>>>>> units, and a data path multiplexer (to scaler units, to ISP, ...).
>>>>>>>
>>>>>>> I understand both RK3568 and RK3588 include an ISP. Do you have insight on
>>>>>>> how would this work, should the support for the ISP be added later on?
>>>>>>
>>>>>> Short answer: Yes and yes.
>>>>>>
>>>>>> Long answer:
>>>>>>
>>>>>> The patch series at hand adds support for the PX30 VIP and the RK3568
>>>>>> VICAP. I cannot really say something about the PX30, but on the RK3568
>>>>>> VICAP and ISP are orthogonal (the ISP features its own MIPI CSI-2
>>>>>> receiver, different from that introduced in this series). Thus, ISP
>>>>>> support can be introduced anytime (whenever someone is motivated ;-)).
>>>>>
>>>>> Won't they both be connected to the same sensor though, and probably the
>>>>> same D-PHY in the SoC ? They don't seem entirely separate to me.
>>>>
>>>> The MIPI CSI-2 DPHY is shared, indeed. Thus, they *maybe technically
>>>> could be* connected to the same sensor, but I don't know whether that
>>>> works and fail to see why anyone would to such a thing (if it is about
>>>> raw capture, the MIPI CSI-2 receiver in the ISP can do that on its own).
>>>>
>>>> The DPHY can be operated in split mode, with two lanes for VICAP and two
>>>> lanes for ISP. This is not implemented yet, but can be done at a later
>>>> stage on PHY level (not media related). In this case, ISP and VICAP can
>>>> receive data from different subdevices via CSI-2.
>>>
>>> The two would be part of the same media graph in that case and as there are
>>> two CSI-2 receivers and a single PHY, the PHY would probably need to have a
>>> sub-device as well, to allow link configuration to be used to select where
>>> the PHY is connected.
>>>
>>> I don't think we have such a setup elsewhere, and supporting this would
>>> require changes in the MC framework.
>>
>> What follows is a response that also addresses issues raised during our
>> off-list discussion.
>>
>> First of all, I agree with you that the RK3568 HW is "a bit special" (to
>> say the least) in that regard. Let's have an outlook to newer SoCs, such
>> as the RK3588: Here, the MIPI CSI-2 DPHYs (there are two of them) with
>> their split mode are present as well, but the assignment is fixed. For
>> example, the RK3588 VICAP has six MIPI CSI-2 receiver units and six MIPI
>> CSI-2 capture units. Units 1 and 2 handle a different MIPI PHY, units 3
>> and 5 handle the DPHYs (without split mode), units 4 and/or 6 are active
>> whenever DPHY 1 and/or 2 is in split mode.
>>
>> I would model this by adding support for more than one (logical) PHYs
>> (phy-cells = <1>;) and assigning the logical PHYs to the MIPI CSI-2
>> receivers. There is not really a possibility to route anything at this
>> point (routing is done in a MUX unit that takes the different MIPI CSI-2
>> receivers as inputs).
>>
>> Now back to the peculiar RK3568 situation: By default the split mode of
>> the DPHY is off and both VICAP and ISP are able to receive the same data
>> (from up to four lanes) with their MIPI CSI-2 receivers (not sure
>> whether both can be active at the same time, though).
> 
> A common use case for capturing data from the same sensor through both
> ISP and VICAP would be routing image data to the ISP and embedded data
> to VICAP. Assuming the hardware is able to do this, there will be users
> interested in this feature.

VICAP and ISP would capture different CSI-2 links. AFAIK the embedded
data is usually transmitted on the same CSI-2 link but on a different
virtual channel. The MIPI CSI receiver in the ISP is capable of
capturing (up to three, I think) virtual channels and passing e.g. one
VC to the ISP and capturing another VC to memory.

>> There are two bits
>> in the GRF that define the lanes that ISP and VICAP receive in split
>> mode (lane 0/1 or lane 2/3). Not sure whether these bits are supposed to
>> be changed during runtime.
>>
>> I would suggest modelling this on PHY level in DT, e.g., by passing
>> reasonable properties to the dphy node, such as
>>     rockchip,dphy-split-mode;
> 
> Split mode should indeed be conveyed through DT, as it's a property of
> the hardware.

Ack!

> 
>>     rockchip,dphy-split-invert;
>> where the former activates the split mode and assigns lanes 0/1 to the
>> ISP and lanes 2/3 to the VICAP, and the latter inverts this assignment
>> (lanes 2/3 to the ISP and lanes 0/1 to the VICAP). This would facilitate
>> the reasonable use cases with reasonable effort.
> 
> This I'm less convinced about. The routing should be dynamic.

Ack.

> How do you envision the connections to the CSI-2 sources to be modelled
> in DT ? We need ports and endpoints, and data-lanes properties. Where
> would those reside on the RK3568 side ?

Goto ASCII sketch below for that...

>> Otherwise, to keep it perfectly general and most flexible and
>> everything, we would have to introduce another subdevice indeed, which
>> would be active on the RK3568 exclusively. Therefore, I don't see that
>> the PHY driver introduces this subdevice, but a specialized (syscon?)
>> MUX driver that deals with the RK3568 GRF bits. Something like this
>>
>>                            |----------------------|     |-------------|
>>  Sensor A --- /2 lanes --- | lane 0/1      to ISP | --- | ISP MIPI RX |
>>                            |                      |     |-------------|
>>                            |                      |
>>                            |                      |     |-------------|
>>  Sensor B --- /2 lanes --- | lane 2/3    to VICAP | --- |VICAP MIPI RX|
>>                            |----------------------|     |-------------|
>>
>> But IMHO this will be too much effort for corner use case that I doubt
>> anyone will actually use.
>>
>> What do you think:
>>  - Let's keep the PHYs out of V4L2/MC, ok?
>>  - Let's model the reasonable use cases with device tree properties in
>>    the dphy DT node, ok?
> 
> I think the routing should be dynamic.

In this case, my suggestion would be the following:

 - An additional MUX subdevice on the RK3568 only, see sketch above
 - One or two input ports (depending on split mode being activated)
 - Two output ports (to ISP MIPI RX, to VICAP MIPI RX)
 - Routing possible 1:N (both input ports can be connected to one or two
   output ports)
 - The subdevice will have ports, endpoints, data-lanes properties on
   both sides.

The nice thing about it is that the current state can be extended once
split mode is implemented, IOW patches 01/18 - 17/18 can be readily applied.

Patch 18/18 must not be applied at this time, as this would be the first
actual usage of the MIPI CSI-2 pipeline. I guess we need to implement
and insert the new subdevice first for the sake of uAPI compatibility.

So pretty-please-with-sugar-on-top can we merge 01-17 so that I don't
have to drag these patches along anymore?

Best regards,
Michael

> 
>>> How does the media graph look like for the device at the moment?
>>
>> Please take a look at the media graph in the documentation patch (PATCH
>> v14 01/18). This is without the ISP, but gives an overview of what the
>> RK3568 VICAP is capable of.
>>
>> Best regards,
>> Michael
>>
>>>> BTW the ISP is able to process the data captured by VICAP, but
>>>> apparently this includes a RAM round trip (VICAP captures to memory, ISP
>>>> operates in mem2mem mode).
>>>>
>>>>> A block diagram that shows connections between the CSI-2 pins, D-PHY,
>>>>> CSI-2 receivers, VICAP and ISP could help.
>>>>>
>>>>>> Once this patch series is merged, I'll push out changes that introduce
>>>>>> support for the RK3588 VICAP. We can discuss the integration of any
>>>>>> RK3588 ISP in this scope then -- and there may be some things to discuss
>>>>>> as there the VICAP and the ISP(s) are directly connected by means of a
>>>>>> MUX unit in the VICAP.
>>>>>>
>>>>>> Alright?
> 


_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip