From mboxrd@z Thu Jan 1 00:00:00 1970 From: Oleksandr Andrushchenko Subject: Re: [PATCH 1/1] cameraif: add ABI for para-virtual camera Date: Tue, 4 Sep 2018 09:56:17 +0300 Message-ID: <53189190-ffe0-9795-b01c-01de8db83acb@gmail.com> References: <20180731093142.3828-1-andr2000@gmail.com> <20180731093142.3828-2-andr2000@gmail.com> <99cd131d-85ae-bbfb-61ef-fdc0401727f6@suse.com> <5505e5af-5b64-b317-a0d8-09c11317926f@gmail.com> <345d7ec3-3ca3-e8fe-28a0-ba299196b5e4@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Return-path: Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1fx5GA-0005wR-G0 for xen-devel@lists.xenproject.org; Tue, 04 Sep 2018 06:56:22 +0000 Received: by mail-lf1-x144.google.com with SMTP id q13-v6so2069574lfc.2 for ; Mon, 03 Sep 2018 23:56:20 -0700 (PDT) In-Reply-To: Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" To: Hans Verkuil , Juergen Gross , xen-devel@lists.xenproject.org, konrad.wilk@oracle.com, boris.ostrovsky@oracle.com, mchehab@kernel.org, linux-media@vger.kernel.org, sakari.ailus@linux.intel.com, koji.matsuoka.xm@renesas.com Cc: Artem Mygaiev , Oleksandr Andrushchenko List-Id: xen-devel@lists.xenproject.org T24gMDkvMDMvMjAxOCAwNjoyNSBQTSwgSGFucyBWZXJrdWlsIHdyb3RlOgo+IEhpIE9sZWtzYW5k ciwKPgo+IE9uIDA5LzAzLzIwMTggMTI6MTYgUE0sIE9sZWtzYW5kciBBbmRydXNoY2hlbmtvIHdy b3RlOgo+PiBPbiAwOC8yMS8yMDE4IDA4OjU0IEFNLCBPbGVrc2FuZHIgQW5kcnVzaGNoZW5rbyB3 cm90ZToKPj4+IE9uIDA4LzE0LzIwMTggMTE6MzAgQU0sIEp1ZXJnZW4gR3Jvc3Mgd3JvdGU6Cj4+ Pj4gT24gMzEvMDcvMTggMTE6MzEsIE9sZWtzYW5kciBBbmRydXNoY2hlbmtvIHdyb3RlOgo+Pj4+ PiBGcm9tOiBPbGVrc2FuZHIgQW5kcnVzaGNoZW5rbyA8b2xla3NhbmRyX2FuZHJ1c2hjaGVua29A ZXBhbS5jb20+Cj4+Pj4+Cj4+Pj4+IFRoaXMgaXMgdGhlIEFCSSBmb3IgdGhlIHR3byBoYWx2ZXMg b2YgYSBwYXJhLXZpcnR1YWxpemVkCj4+Pj4+IGNhbWVyYSBkcml2ZXIgd2hpY2ggZXh0ZW5kcyBY ZW4ncyByZWFjaCBtdWx0aW1lZGlhIGNhcGFiaWxpdGllcyBldmVuCj4+Pj4+IGZhcnRoZXIgZW5h YmxpbmcgaXQgZm9yIHZpZGVvIGNvbmZlcmVuY2luZywgSW4tVmVoaWNsZSBJbmZvdGFpbm1lbnQs Cj4+Pj4+IGhpZ2ggZGVmaW5pdGlvbiBtYXBzIGV0Yy4KPj4+Pj4KPj4+Pj4gVGhlIGluaXRpYWwg Z29hbCBpcyB0byBzdXBwb3J0IG1vc3QgbmVlZGVkIGZ1bmN0aW9uYWxpdHkgd2l0aCB0aGUKPj4+ Pj4gZmluYWwgaWRlYSB0byBtYWtlIGl0IHBvc3NpYmxlIHRvIGV4dGVuZCB0aGUgcHJvdG9jb2wg aWYgbmVlZCBiZToKPj4+Pj4KPj4+Pj4gMS4gUHJvdmlkZSBtZWFucyBmb3IgYmFzZSB2aXJ0dWFs IGRldmljZSBjb25maWd1cmF0aW9uOgo+Pj4+PiAgICAtIHBpeGVsIGZvcm1hdHMKPj4+Pj4gICAg LSByZXNvbHV0aW9ucwo+Pj4+PiAgICAtIGZyYW1lIHJhdGVzCj4+Pj4+IDIuIFN1cHBvcnQgYmFz aWMgY2FtZXJhIGNvbnRyb2xzOgo+Pj4+PiAgICAtIGNvbnRyYXN0Cj4+Pj4+ICAgIC0gYnJpZ2h0 bmVzcwo+Pj4+PiAgICAtIGh1ZQo+Pj4+PiAgICAtIHNhdHVyYXRpb24KPj4+Pj4gMy4gU3VwcG9y dCBzdHJlYW1pbmcgY29udHJvbAo+Pj4+PiA0LiBTdXBwb3J0IHplcm8tY29weWluZyB1c2UtY2Fz ZXMKPj4+Pj4KPj4+Pj4gU2lnbmVkLW9mZi1ieTogT2xla3NhbmRyIEFuZHJ1c2hjaGVua28KPj4+ Pj4gPG9sZWtzYW5kcl9hbmRydXNoY2hlbmtvQGVwYW0uY29tPgo+Pj4+IFNvbWUgc3R5bGUgaXNz dWVzIGJlbG93Li4uCj4+PiBXaWxsIGZpeCBhbGwgdGhlIGJlbG93LCB0aGFuayB5b3UhCj4+Pgo+ Pj4gSSB3b3VsZCBsaWtlIHRvIGRyYXcgc29tZSBhdHRlbnRpb24gb2YgdGhlIExpbnV4L1Y0TCBj b21tdW5pdHkgdG8gdGhpcwo+Pj4gcHJvdG9jb2wgYXMgdGhlIHBsYW4gaXMgdGhhdCBvbmNlIGl0 IGlzIGFjY2VwdGVkIGZvciBYZW4gd2UgcGxhbiB0bwo+Pj4gdXBzdHJlYW0gYSBMaW51eCBjYW1l cmEgZnJvbnQtZW5kIGtlcm5lbCBkcml2ZXIgd2hpY2ggd2lsbCBiZSBiYXNlZAo+Pj4gb24gdGhp cyB3b3JrIGFuZCB3aWxsIGJlIGEgVjRMMiBkZXZpY2UgZHJpdmVyICh0aGlzIGlzIHdoeSBJIGhh dmUgc2VudAo+Pj4gdGhpcyBwYXRjaCBub3Qgb25seSB0byBYZW4sIGJ1dCB0byB0aGUgY29ycmVz cG9uZGluZyBMaW51eCBtYWlsaW5nIGxpc3QKPj4+IGFzIHdlbGwpCj4+IHBpbmcKPiBTb3JyeSwg dGhpcyBnb3QgYnVyaWVkIGluIG15IG1haWxib3gsIEkgb25seSBjYW1lIGFjcm9zcyBpdCB0b2Rh eS4gSSdsbCB0cnkKPiB0byByZXZpZXcgdGhpcyB0aGlzIHdlZWssIGlmIG5vdCwganVzdCBwaW5n IG1lIGFnYWluLgpUaGFuayB5b3UgZm9yIHlvdXIgdGltZQo+Cj4gSSBoYWQgb25lIGhpZ2gtbGV2 ZWwgcXVlc3Rpb24sIHRob3VnaDoKPgo+IFdoYXQgdHlwZXMgb2YgaGFyZHdhcmUgZG8geW91IGlu dGVuZCB0byB0YXJnZXQ/IFRoaXMgaW5pdGlhbCB2ZXJzaW9uIHRhcmdldHMKPiAodmVyeSkgc2lt cGxlIHdlYmNhbXMsIGJ1dCB3aGF0IGFib3V0IEhETUkgb3IgU0RUViByZWNlaXZlcnM/IE9yIGhh cmR3YXJlCj4gY29kZWNzPyBPciBjb21wbGV4IGVtYmVkZGVkIHZpZGVvIHBpcGVsaW5lcz8KPgo+ IEluIG90aGVyIHdvcmRzLCB3aGVyZSBhcmUgeW91IHBsYW5uaW5nIHRvIGRyYXcgdGhlIGxpbmU/ Cj4KPiBFdmVuIHdpdGgganVzdCBzaW1wbGUgY2FtZXJhcyB0aGVyZSBpcyBhIGRpZmZlcmVuY2Ug YmV0d2VlbiByZWd1bGFyIFVWQwo+IHdlYmNhbXMgYW5kIGNhbWVyYXMgdXNlZCB3aXRoIGVtYmVk ZGVkIHN5c3RlbXM6IGZvciB0aGUgbGF0dGVyIHlvdSBvZnRlbgo+IG5lZWQgdG8gcHJvdmlkZSBt b3JlIGNvbnRyb2wgdy5yLnQuIHdoaXRlLWJhbGFuY2luZyBldGMuLCB0aGluZ3MgdGhhdCBhCj4g VVZDIHdlYmNhbSB3aWxsIGdlbmVyYWxseSBkbyBmb3IgeW91IGluIHRoZSB3ZWJjYW0ncyBmaXJt d2FyZS4KVGhlIHVzZS1jYXNlcyB3ZSB3YW50IHRvIGltcGxlbWVudCBhcmUgbW9zdGx5IGluIGF1 dG9tb3RpdmUvZW1iZWRkZWQgZG9tYWluLApzbyB0aGVyZSBhcmUgbWFueSBwZXJmb3JtYW5jZSBy ZXN0cmljdGlvbnMgYXBwbHkuCldlIGFyZSBub3QgdGFyZ2V0aW5nIHZpcnR1YWxpemluZyB2ZXJ5 IGNvbXBsZXggaGFyZHdhcmUgYW5kIGhhdmUgbm8gCmludGVudGlvbgp0byBtYWtlIGEgMToxIG1h cHBpbmcgb2YgdGhlIHJlYWwgaGFyZHdhcmU6IGZvciB0aGF0IG9uZSBjYW4gcGFzcy10aHJvdWdo CmEgcmVhbCBIVyBkZXZpY2UgdG8gYSB2aXJ0dWFsIG1hY2hpbmUgKFZNKS4gVGhlIGdvYWwgaXMg dG8gc2hhcmUgYSBzaW5nbGUKY2FtZXJhIGRldmljZSB0byBtdWx0aXBsZSB2aXJ0dWFsIG1hY2hp bmVzLCBubyBjb2RlY3MsIHJlY2VpdmVycyBldGMuCgpDb250cm9sbGluZyB0aGUgc2FtZSBIVyBk ZXZpY2UgZnJvbSBkaWZmZXJlbnQgVk1zIGRvZXNuJ3QgbG9vayBmZWFzaWJsZToKd2hhdCBpZiB0 aGUgc2FtZSBjb250cm9sIGlzIHNldCB0byBkaWZmZXJlbnQgdmFsdWVzIGZyb20gZGlmZmVyZW50 IFZNcz8KT2YgY291cnNlLCB0aGlzIGNhbiBiZSBhY2hpZXZlZCBpZiB0aGUgY29ycmVzcG9uZGlu ZyBiYWNrZW5kIGNhbiAKcG9zdC1wcm9jZXNzCm9yaWdpbmFsIGNhbWVyYSBpbWFnZSB3aXRoIEdQ VSwgZm9yIGV4YW1wbGUsIHRodXMgYXBwbHlpbmcgZGlmZmVyZW50IGZpbHRlcnMKZm9yIGRpZmZl cmVudCBWTXMgZWZmZWN0aXZlbHkgZW11bGF0aW5nIGNhbWVyYSBjb250cm9scy4KQnV0IHRoaXMg cmVxdWlyZXMgYWRkaXRpb25hbCBDUFUvR1BVIHBvd2VyIHdoaWNoIHdlIHRyeSB0byBhdm9pZC4K ClN5c3RlbSBwYXJ0aXRpb25pbmcgKGNhbWVyYSBhbmQgY29udHJvbHMgYXNzaWdubWVudCkgaXMg ZG9uZSBhdCAKY29uZmlndXJhdGlvbgp0aW1lIChyZW1lbWJlciB3ZSBhcmUgaW4gYXV0b21vdGl2 ZS9lbWJlZGRlZCB3b3JsZCwgc28gbW9zdCBvZiB0aGUgdGltZSAKdGhlIHNldApvZiBWTXMgcmVx dWlyaW5nIGNhbWVyYXMgaXMga25vd24gYXQgdGhpcyBzdGFnZSBhbmQgdGhlIGNvbmZpZ3VyYXRp b24gCnJlbWFpbnMKc3RhdGljIGF0IHJ1bi10aW1lKS4gU28sIHdoZW4gcGFyYS12aXJ0dWFsaXpl ZCAoUFYpIGFwcHJvYWNoIGlzIHVzZWQgdGhlbiB3ZQpvbmx5IGltcGxlbWVudCB2ZXJ5IGJhc2lj IGNvbnRyb2xzICh0aG9zZSBmb3VuZCBpbiB0aGUgcHJvdG9jb2wpLCBzbyBvbmUgY2FuCmFzc2ln biBzZXQgb2YgY29udHJvbHMgKGFsbCBvciBzb21lKSB0byBvbmUgb2YgdGhlIFZNcyAobWFpbiBv ciBtaXNzaW9uIApjcml0aWNhbApWTSBvciB3aGF0ZXZlcikgYWxsb3dpbmcgdGhhdCBWTSB0byBh ZGp1c3RzIHRob3NlIGZvciBhbGwgVk1zIGF0IG9uY2UuIApGb3Igb3RoZXIKVk1zIHRoaW5rIG9m IGl0IGFzIGZpcm13YXJlIGltcGxlbWVudGVkIGFkanVzdG1lbnQuIEFuZCB0aGUgYmFja2VuZCBz dGlsbApjb250cm9scyB0aGUgcmVzdCBvZiB0aGUgY29udHJvbHMgb2YgdGhlIHJlYWwgSFcgY2Ft ZXJhIHlvdSBtZW50aW9uLgoKSnVzdCBhbiBleGFtcGxlIG9mIGF1dG9tb3RpdmUgdXNlLWNhc2Ug KHdlIGNhbiBpbWFnaW5lIG1hbnkgbW9yZSk6CjEuIERyaXZlciBEb21haW4gLSBvd25zIHJlYWwg Y2FtZXJhIEhXIGFuZCBydW5zIHRoZSBjYW1lcmEgYmFja2VuZC4KIMKgwqAgVXNlcyBjYW1lcmEg b3V0cHV0IGZvciBtaXNzaW9uIGNyaXRpY2FsIHRhc2tzLCBlLmcuIHBhcmtpbmcgYXNzaXN0YW5j ZS4KMi4gSW4tVmVoaWNsZSBJbmZvdGFpbm1lbnQgZG9tYWluIC0gdXNlcyBQViBjYW1lcmEgZm9y IGluZm90YWlubWVudCAKcHVycG9zZXMsCiDCoMKgIGUuZy4gdGFraW5nIHBpY3R1cmVzIHdoaWxl IGluIG1vdGlvbi4KMy4gTmF2aWdhdGlvbiBkb21haW4gLSB1c2VzIFBWIGNhbWVyYSBmb3IgaGln aCBkZWZpbml0aW9uIG1hcHMKCkhvcGUsIHRoaXMgaGVscHMgdW5kZXJzdGFuZGluZyB0aGUgcG9z c2libGUgdXNlcyBvZiB0aGUgcHJvcG9zZWQgCnByb3RvY29sLCBpdHMKaW50ZW50aW9uIGFuZCBy ZXN0cmljdGlvbnMuCgo+IFJlZ2FyZHMsCj4KPiAJSGFucwpUaGFuayB5b3UsCk9sZWtzYW5kcgoK X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVucHJvamVjdC5vcmcKaHR0cHM6Ly9saXN0 cy54ZW5wcm9qZWN0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3hlbi1kZXZlbA== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mail-lf1-f65.google.com ([209.85.167.65]:39538 "EHLO mail-lf1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726061AbeIDLUE (ORCPT ); Tue, 4 Sep 2018 07:20:04 -0400 Received: by mail-lf1-f65.google.com with SMTP id v77-v6so2047786lfa.6 for ; Mon, 03 Sep 2018 23:56:20 -0700 (PDT) Subject: Re: [Xen-devel][PATCH 1/1] cameraif: add ABI for para-virtual camera To: Hans Verkuil , Juergen Gross , xen-devel@lists.xenproject.org, konrad.wilk@oracle.com, boris.ostrovsky@oracle.com, mchehab@kernel.org, linux-media@vger.kernel.org, sakari.ailus@linux.intel.com, koji.matsuoka.xm@renesas.com Cc: Oleksandr Andrushchenko , Artem Mygaiev References: <20180731093142.3828-1-andr2000@gmail.com> <20180731093142.3828-2-andr2000@gmail.com> <99cd131d-85ae-bbfb-61ef-fdc0401727f6@suse.com> <5505e5af-5b64-b317-a0d8-09c11317926f@gmail.com> <345d7ec3-3ca3-e8fe-28a0-ba299196b5e4@gmail.com> From: Oleksandr Andrushchenko Message-ID: <53189190-ffe0-9795-b01c-01de8db83acb@gmail.com> Date: Tue, 4 Sep 2018 09:56:17 +0300 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Sender: linux-media-owner@vger.kernel.org List-ID: On 09/03/2018 06:25 PM, Hans Verkuil wrote: > Hi Oleksandr, > > On 09/03/2018 12:16 PM, Oleksandr Andrushchenko wrote: >> On 08/21/2018 08:54 AM, Oleksandr Andrushchenko wrote: >>> On 08/14/2018 11:30 AM, Juergen Gross wrote: >>>> On 31/07/18 11:31, Oleksandr Andrushchenko wrote: >>>>> From: Oleksandr Andrushchenko >>>>> >>>>> This is the ABI for the two halves of a para-virtualized >>>>> camera driver which extends Xen's reach multimedia capabilities even >>>>> farther enabling it for video conferencing, In-Vehicle Infotainment, >>>>> high definition maps etc. >>>>> >>>>> The initial goal is to support most needed functionality with the >>>>> final idea to make it possible to extend the protocol if need be: >>>>> >>>>> 1. Provide means for base virtual device configuration: >>>>> - pixel formats >>>>> - resolutions >>>>> - frame rates >>>>> 2. Support basic camera controls: >>>>> - contrast >>>>> - brightness >>>>> - hue >>>>> - saturation >>>>> 3. Support streaming control >>>>> 4. Support zero-copying use-cases >>>>> >>>>> Signed-off-by: Oleksandr Andrushchenko >>>>> >>>> Some style issues below... >>> Will fix all the below, thank you! >>> >>> I would like to draw some attention of the Linux/V4L community to this >>> protocol as the plan is that once it is accepted for Xen we plan to >>> upstream a Linux camera front-end kernel driver which will be based >>> on this work and will be a V4L2 device driver (this is why I have sent >>> this patch not only to Xen, but to the corresponding Linux mailing list >>> as well) >> ping > Sorry, this got buried in my mailbox, I only came across it today. I'll try > to review this this week, if not, just ping me again. Thank you for your time > > I had one high-level question, though: > > What types of hardware do you intend to target? This initial version targets > (very) simple webcams, but what about HDMI or SDTV receivers? Or hardware > codecs? Or complex embedded video pipelines? > > In other words, where are you planning to draw the line? > > Even with just simple cameras there is a difference between regular UVC > webcams and cameras used with embedded systems: for the latter you often > need to provide more control w.r.t. white-balancing etc., things that a > UVC webcam will generally do for you in the webcam's firmware. The use-cases we want to implement are mostly in automotive/embedded domain, so there are many performance restrictions apply. We are not targeting virtualizing very complex hardware and have no intention to make a 1:1 mapping of the real hardware: for that one can pass-through a real HW device to a virtual machine (VM). The goal is to share a single camera device to multiple virtual machines, no codecs, receivers etc. Controlling the same HW device from different VMs doesn't look feasible: what if the same control is set to different values from different VMs? Of course, this can be achieved if the corresponding backend can post-process original camera image with GPU, for example, thus applying different filters for different VMs effectively emulating camera controls. But this requires additional CPU/GPU power which we try to avoid. System partitioning (camera and controls assignment) is done at configuration time (remember we are in automotive/embedded world, so most of the time the set of VMs requiring cameras is known at this stage and the configuration remains static at run-time). So, when para-virtualized (PV) approach is used then we only implement very basic controls (those found in the protocol), so one can assign set of controls (all or some) to one of the VMs (main or mission critical VM or whatever) allowing that VM to adjusts those for all VMs at once. For other VMs think of it as firmware implemented adjustment. And the backend still controls the rest of the controls of the real HW camera you mention. Just an example of automotive use-case (we can imagine many more): 1. Driver Domain - owns real camera HW and runs the camera backend.    Uses camera output for mission critical tasks, e.g. parking assistance. 2. In-Vehicle Infotainment domain - uses PV camera for infotainment purposes,    e.g. taking pictures while in motion. 3. Navigation domain - uses PV camera for high definition maps Hope, this helps understanding the possible uses of the proposed protocol, its intention and restrictions. > Regards, > > Hans Thank you, Oleksandr