Re: multi-sensor media controller

[Jeremy Gebben <jgebben@codeaurora.org>]

Guennadi,


On 5/29/16 8:43 AM, Guennadi Liakhovetski wrote:
> Hi Jeremy,
>
> On Fri, 27 May 2016, Jeremy Gebben wrote:
>
>> Hi,
>>
>> Can someone give me a quick sanity check on a media controller set up?
>>
>> We have have devices (well, phones) that can have 2 or more sensors and 2 or
>> more 'front end' ISPs.  The ISPs take CSI input from a sensor, and can produce
>> Bayer and YUV output to memory. There is a bridge between the sensors and ISPs
>> which allow any ISP to receive input from any sensor. We would (eventually)
>> like to use the request API to ensure that sensor and ISP settings match for
>> every frame.
>>
>> We use this hardware in several different ways, some of the interesting ones
>> are described below:
>>
>> 1. 2 sensors running independently, and possibly at different frame rates. For
>> example in video chat you might want a "Picture in Picture" setup where you
>> send output from both a front and back sensor.
>> (note: 'B' is the bridge)
>>
>> SENSOR_A --> B --> ISP_X --> BUFFERS
>>              B
>> SENSOR_B --> B --> ISP_Y --> BUFFERS
>>
>> 2. Single sensor, dual ISP. High resolution use of a single sensor may
>> require both ISPs to work on different regions of the image. For example,
>> ISP_X might process the left half of the image while ISP_Y processes the
>> right.
>>
>> SENSOR_A --> B --> ISP_X ----> BUFFERS
>>              B --> ISP_Y --/
>>              B
>>
>> 3. 2 different types of sensors working together to produce a single set of
>> images. Processing must be synchronized, and eventually the buffers from
>> SENSOR_A and SENSOR_C will be combined by other processing not shown here.
>>
>> SENSOR_A --> B --> ISP_X --> BUFFERS
>> SENSOR_C --> B --> ISP_Y --> BUFFERS
>>              B
>>
>> It seems to me that the way to do handle all these cases would be to put all
>> of the hardware in a single media_device:
>>
>>  +----------------------+
>>  |  SENSOR_A  B  ISP_X  |
>>  |  SENSOR_C  B         |
>>  |  SENSOR_B  B  ISP_Y  |
>>  +----------------------+
>>
>> This makes cases #2 and #3 above easy to configure.
>
> Yes, agree.
>
>> But the topology can get
>> rather large if the number of sensors and ISPs goes up.
>
> We've seen some rather large topology graphs already :)
>
>> Also, case #1 could be
>> a little strange because there would be 2 independent sets of requests coming
>> to the same media_device at the same time.
>
> Do you mean, because request API calls are made to the /dev/media0 device?
> I don't remember the details, is this how the API is supposed to be used?
> Isn't there a way to direct the calls to specific /dev/video* devices or
> to subdevices?

See: https://meet.quicinc.com/hosseins/34WMDH75

You allocate a request id using MEDIA_REQ_CMD_ALLOC on a /dev/media* 
device. Then you call /dev/video* ioctls to change settings, passing the 
request id as an argument. Then you call MEDIA_REQ_CMD_QUEUE which 
applies all the settings.

My concern is that having 2 independent camera pipelines using request 
ids from the same /dev/media could cause problems somehow. But I cannot 
quantify "somehow" so I guess I'll look into the patches further to work 
it out.


>> Am I on the right track here?
>>
>> I did find Guennadi's presentation about multi-stream:
>> https://linuxtv.org/downloads/presentations/media_summit_2016_san_diego/v4l2-multistream.pdf
>>
>> ...but I'm not sure I follow all of it from the slides alone, and
>> we don't have the mux/demux hardware that was the focus of his presentation.
>
> Doesn't your bridge also perform the mux-demux role in some
> configurations?

Hmm, yeah, maybe I oversimplified. I probably need to dig in to the 
hardware docs further.


> But that isn't the main point anyway. As a part of a
> solution for the multi-stream set up, the use of stream routing has been
> proposed. So, using that, you might be able to represent your bridge as a
> stream router. Details of a routing API are still to be clarified.

I'll watch for further traffic on the routing API.

Thank you for the feedback.

Jeremy



>
> Thanks
> Guennadi
>
>> Thanks,
>>
>> Jeremy
>>
>> --
>>  The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
>>  a Linux Foundation Collaborative Project
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-media" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>


-- 
  The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
  a Linux Foundation Collaborative Project