Re: [PATCH v12 00/30] v4l: routing and streams support

[Laurent Pinchart <laurent.pinchart@ideasonboard.com>]

On Mon, Aug 08, 2022 at 09:45:38AM +0300, Tomi Valkeinen wrote:
> On 05/08/2022 18:14, Milen Mitkov (Consultant) wrote:
> 
> > >> If I want to configure the CSI decoder subdev (msm_csid0) to receive 1
> > >> stream on the sink pad and route it to 2 streams on its source pad. Is
> > >> my thinking correct?
> > >
> > > Yes, if your HW can do that. I don't have HW that can split (or clone) a
> > > stream, so it's possible that the use case doesn't work.
> > 
> > Now here's the main question. We use the CSI decoder (CSID) hardware to 
> > split
> > one stream from the sensor into 2 or more streams based on datatype or 
> > CSI virtual channel.
> > 
> > Basically, the complete pipeline is something like this, for 2 virtual 
> > channels:
> > 
> >                               -> ISP line 0 -> videodev /dev/video0
> >                             /
> > sensor -> CSIPHY -> CSID ->
> >                             \
> >                               -> ISP line 1 -> videodev /dev/video1
> >                                     
> >                                     
> > So my idea was to make the CSID subdevice multistream API compliant
> > (e.g. V4L2_SUBDEV_FL_STREAMS, manage the active state with the
> > V4L2 API v4l2_subdev_get_locked_active_state, take care of routing setup 
> > etc.),
> > but keep the rest of the devices the way they are.
> 
> That's now how the streams support has been designed. Your sensor 
> provides two streams, and all the drivers that pass through multiple 
> streams needs to be ported to streams API. So in your case, I believe 
> everything but the "ISP line" needs to support streams.

To add a bit of information here, the important thing to understand is
that streams and physical links are two different concepts. The above
diagram describes the physical links (both outside the SoC, and inside
it). Streams are carried by physical links, and a link can carry
multiple streams (hence the name "multiplexed streams" used in this
patch series). If the sensor outputs image data and embedded data with
two CSI-2 DT on one VC, that's two streams carried over the sensor ->
CSIPHY link, and the same two streams going over the CSIPHY -> CSID
link. The CSID demultiplexes the streams, with one stream going to ISP
line 0 and the other one to ISP line 1. As Tomi explained, every subdev
that deals with multiple streams has to implement the new API. This
includes, in this case, the sensor, the CSIPHY and the CSID.

If the sensor were to output two images in different resolutions over
two VCs, it would conceptually be the same, with two streams. If it were
to output image data, embedded data and black level lines over with 3
DTs over one VC, that would be three streams. And so on.

> > The CSID subdev must take 1 stream on the sink pad and output on 2 
> > source pads.
> > 
> > The routing configuration I use for the CSID subdev looks like this:
> > 
> > media-ctl -R '"msm_csid0"[0/0->1/0[0x1],0/0->2/0[0x1]]'
> > 
> > 0 - sink pad, 1 - first source pad, 2 - second source pad
> > 
> > However, this routing setup fails with the validation in 
> > v4l2_link_validate_get_streams().
> > The logic there figures these are duplicate streams because because they 
> > start at the same sink pad.
> > 
> > To summarize my questions:
> > 
> > 1. Is there some sort of restriction that the same sink pad can't be 
> > used for more than 1 stream starting from it?
> 
> In theory no, but it hasn't been tested. I think this case would mainly 
> be cloning of the stream, not really splitting it.
> 
> > 2. Is it ok to migrate only one subdevice to the multistream API
> > or should all possible subdevices in the pipeline be migrated?
> 
> It's ok to mix streams and non-streams subdevices, but the non-streams 
> subdevs must only use a single stream. E.g. You could have 4 non-streams 
> cameras, each providing a single stream to a bridge. The bridge would 
> support streams, and the bridge would send the 4 streams in a single 
> CSI-2 bus.
> 
> Now, that said, I don't think anything strictly prevents from supporting 
> stream splitting, but as I mentioned above, it's not been tested or 
> really even considered very much. It's also a bit ambiguous and unclear 
> and I'd stay away from it if full streams support makes sense.
> 
> I think if a source subdev (sensor) knows that it's providing multiple 
> streams, then it should use streams API to provide those. I.e. if the 
> sensor is providing different types of data, using VCs or DTs, then 
> those are clearly separate streams and the sensor driver must be aware 
> of them.
> 
> Stream splitting might came into play in situations where the sensor 
> provides just a single stream, but a bridge subdev splits it based on 
> information the sensor can't be aware of. For example, the sensor 
> provides a normal pixel stream, and the bridge subdev splits the frames 
> into two halves, sending upper half to output 1 and lower half to output 2.

We've tested splitting on an i.MX8MP, with two different processing
pipelines capturing the stream produced by a single YUV sensor, in
different resolutions and formats. It works (or at least worked with v11
of the streams series, I'll update the code to v13 and retest).

-- 
Regards,

Laurent Pinchart