Re: [PATCH v7 0/4] media: camss: sm8250: Virtual channels support for SM8250

["Milen Mitkov (Consultant)" <quic_mmitkov@quicinc.com>]

On 08/01/2023 04:54, Laurent Pinchart wrote:
> On Sun, Jan 08, 2023 at 04:51:22AM +0200, Laurent Pinchart wrote:
>> Hi Milen,
>>
>> On Fri, Dec 09, 2022 at 11:40:33AM +0200, quic_mmitkov@quicinc.com wrote:
>>> From: Milen Mitkov <quic_mmitkov@quicinc.com>
>>>
>>> For v7:
>>> - Fix an issue with output state for different versions of the IFE
>>>    hardware (for platforms different from QRB5, e.g. QRB3).
>>>
>>> For v6:
>>> - Fix for a potential race condition in csid
>>> - More detailed description on how to use/test this feature in
>>>    user-space in the last patch.
>>>
>>> For v5:
>>> - Use entity->use_count instead of s_stream subdev call ret code to
>>>    check if another instance of the pipeline is running. Prevents an
>>>    error on 6.1 and up, when stopping one of several simultaneous
>>>    instances.
>>> - flush buffers instead of just returning if the pipeline didn't start.
>>>
>>> For v4:
>>> - fixes the warning reported by the kernel test robot
>>> - tiny code change to enable the vc functionality with the partially-applied
>>>    multistream patches on linux-next (tested on tag:next-20221010)
>>>
>>> For v3:
>>> - setting the sink pad format on the CSID entity will now propagate the
>>>    format to the source pads to keep the subdev in a valid internal state.
>>> - code syntax improvements
>>>
>>> For v2:
>>> - code syntax improvements
>>> - The info print for the enabled VCs was demoted to a dbg print. Can be
>>>    enabled with dynamic debug, e.g.:
>>> echo "file drivers/media/platform/qcom/camss/* +p" > /sys/kernel/debug/dynamic_debug/control
>>>
>>> NOTE: These changes depend on the multistream series, that as of yet
>>> is still not merged upstream. However, part of the
>>> multistream patches are merged in linux-next (tested on
>>> tag:next-20221010), including the patch that introduces the
>>> video_device_pipeline_alloc_start() functionality. This allows
>>> applying and using this series on linux-next without applying the
>>> complete multistream set.
>>>
>>> The CSID hardware on SM8250 can demux the input data stream into
>>> maximum of 4 multiple streams depending on virtual channel (vc)
>>> or data type (dt) configuration.
>>>
>>> Situations in which demuxing is useful:
>>> - HDR sensors that produce a 2-frame HDR output, e.g. a light and a dark frame
>>>    (the setup we used for testing, with the imx412 sensor),
>>>    or a 3-frame HDR output - light, medium-lit, dark frame.
>>> - sensors with additional metadata that is streamed over a different
>>>    virtual channel/datatype.
>>> - sensors that produce frames with multiple resolutions in the same pixel
>>>    data stream
>>>
>>> With these changes, the CSID entity has, as it did previously, a single
>>> sink port (0), and always exposes 4 source ports (1, 2,3, 4). The
>>> virtual channel configuration is determined by which of the source ports
>>> are linked to an output VFE line.
> Another question, how does this work when demultiplexing different data
> types for the same virtual channel ?
>
>>> For example, the link below will
>>> configure the CSID driver to enable vc 0 and vc 1:
>>>
>>> media-ctl -l '"msm_csid0":1->"msm_vfe0_rdi0":0[1]'
>>> media-ctl -l '"msm_csid0":2->"msm_vfe0_rdi1":0[1]'
>>>
>>> which will be demuxed and propagated into /dev/video0
>>> and /dev/video1 respectively. With this, the userspace can use
>>> any normal V4L2 client app to start/stop/queue/dequeue from these
>>> video nodes. Tested with the yavta app.
>> I'd like to get a high-level view of the result. Could you provide the
>> media graph with this series applied (both -p and --print-dot) ?
>>
>>> The format of each RDI channel of the used VFE(s) (msm_vfe0_rdi0,
>>> msm_vfe0_rdi1,...) must also be configured explicitly.
>>>
>>> Note that in order to keep a valid internal subdevice state,
>>> setting the sink pad format of the CSID subdevice will propagate
>>> this format to the source pads. However, since the CSID hardware
>>> can demux the input stream into several streams each of which can
>>> be a different format, in that case each source pad's
>>> format must be set individually, e.g.:
>>>
>>> media-ctl -V '"msm_csid0":1[fmt:SRGGB10/3840x2160]'
>>> media-ctl -V '"msm_csid0":2[fmt:SRGGB10/960x540]'
>>>
>>> Milen Mitkov (4):
>>>    media: camss: sm8250: Virtual channels for CSID
>>>    media: camss: vfe: Reserve VFE lines on stream start and link to CSID
>>>    media: camss: vfe-480: Multiple outputs support for SM8250
>>>    media: camss: sm8250: Pipeline starting and stopping for multiple
>>>      virtual channels
>>>
>>>   .../platform/qcom/camss/camss-csid-gen2.c     | 54 ++++++++++------
>>>   .../media/platform/qcom/camss/camss-csid.c    | 44 +++++++++----
>>>   .../media/platform/qcom/camss/camss-csid.h    | 11 +++-
>>>   .../media/platform/qcom/camss/camss-vfe-170.c |  4 +-
>>>   .../media/platform/qcom/camss/camss-vfe-480.c | 61 ++++++++++++-------
>>>   .../platform/qcom/camss/camss-vfe-gen1.c      |  4 +-
>>>   drivers/media/platform/qcom/camss/camss-vfe.c |  1 +
>>>   .../media/platform/qcom/camss/camss-video.c   | 21 ++++++-
>>>   drivers/media/platform/qcom/camss/camss.c     |  2 +-
>>>   9 files changed, 138 insertions(+), 64 deletions(-)

Hi Laurent,

Happy New Year!

Would be good if you can help us merge this series in the upcoming merge 
window. I will answer your 3 questions here.

Q: I'd like to get a high-level view of the result. Could you provide 
the media graph with this series applied (both -p and --print-dot) ?

A: As I am not sure what is the policy with regards to mail attachments 
on the kernel mailing lists I've base64 encoded a small archive that has 
4 different media-ctl dumps: -p before and after media links setup (for 
2 virtual channels demuxing on CSID0) and --print-dot before and after 
media links setup. It can be found here:

https://pastes.io/sp0pkgrgfh

Copy/paste the content to a file and: base64 -d <file> | tar -gz

p.s. if there's another way to share files here please mention it :)


Q: How does this work when demultiplexing different data types for the 
same virtual channel ?

A: Our open-source customers defined the scope of work and they needed 
virtual channels demuxing support only. Demuxing via datatype is not 
supported currently.


Q: I was wondering if Qualcomm had a spare RB5 vision development kit 
that I could use for testing, both for this series, and for future 
development (including the libcamera side).

A: We have to check internally as there's a limited supply of RB5 
boards. We'll get back to you.


Thanks,

Milen