From mboxrd@z Thu Jan 1 00:00:00 1970 From: Laurent Pinchart Subject: Re: DRM Format Modifiers in v4l2 Date: Thu, 31 Aug 2017 17:28:54 +0300 Message-ID: <4559442.sz5HF0f0o4@avalon> References: <20170821155203.GB38943@e107564-lin.cambridge.arm.com> <47128f36-2990-bd45-ead9-06a31ed8cde0@xs4all.nl> <20170824111430.GB25711@e107564-lin.cambridge.arm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from galahad.ideasonboard.com (galahad.ideasonboard.com [185.26.127.97]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4F0D36E72A for ; Thu, 31 Aug 2017 14:28:37 +0000 (UTC) In-Reply-To: <20170824111430.GB25711@e107564-lin.cambridge.arm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Brian Starkey Cc: Hans Verkuil , jonathan.chai@arm.com, dri-devel , "linux-media@vger.kernel.org" List-Id: dri-devel@lists.freedesktop.org SGkgQnJpYW4sCgpPbiBUaHVyc2RheSwgMjQgQXVndXN0IDIwMTcgMTQ6MTQ6MzEgRUVTVCBCcmlh biBTdGFya2V5IHdyb3RlOgo+IE9uIE1vbiwgQXVnIDIxLCAyMDE3IGF0IDA2OjM2OjI5UE0gKzAy MDAsIEhhbnMgVmVya3VpbCB3cm90ZToKPiA+IE9uIDA4LzIxLzIwMTcgMDY6MDEgUE0sIERhbmll bCBWZXR0ZXIgd3JvdGU6Cj4gPj4gT24gTW9uLCBBdWcgMjEsIDIwMTcgYXQgNTo1MiBQTSwgQnJp YW4gU3RhcmtleSB3cm90ZToKPiA+Pj4gSGkgYWxsLAo+ID4+PiAKPiA+Pj4gSSBjb3VsZG4ndCBm aW5kIHRoaXMgdG9waWMgdGFsa2VkIGFib3V0IGVsc2V3aGVyZSwgYnV0IGFwb2xvZ2llcyBpZgo+ ID4+PiBpdCdzIGEgZHVwbGljYXRlIC0gSSdsbCBiZSBnbGFkIHRvIGJlIHN0ZWVyZWQgaW4gdGhl IGRpcmVjdGlvbiBvZiBhCj4gPj4+IHRocmVhZC4KPiA+Pj4gCj4gPj4+IFdlJ2QgbGlrZSB0byBz dXBwb3J0IERSTSBmb3JtYXQgbW9kaWZpZXJzIGluIHY0bDIgaW4gb3JkZXIgdG8gc2hhcmUKPiA+ Pj4gdGhlIGRlc2NyaXB0aW9uIG9mIGRpZmZlcmVudCAobW9zdGx5IHByb3ByaWV0YXJ5KSBidWZm ZXIgZm9ybWF0cwo+ID4+PiBiZXR3ZWVuIGUuZy4gYSB2NGwyIGRldmljZSBhbmQgYSBEUk0gZGV2 aWNlLgo+ID4+PiAKPiA+Pj4gRFJNIGZvcm1hdCBtb2RpZmllcnMgYXJlIGRlZmluZWQgaW4gaW5j bHVkZS91YXBpL2RybS9kcm1fZm91cmNjLmggYW5kCj4gPj4+IGFyZSBhIHZlbmRvci1uYW1lc3Bh Y2VkIDY0LWJpdCB2YWx1ZSB1c2VkIHRvIGRlc2NyaWJlIHZhcmlvdXMKPiA+Pj4gdmVuZG9yLXNw ZWNpZmljIGJ1ZmZlciBsYXlvdXRzLiBUaGV5IGFyZSBjb21iaW5lZCB3aXRoIGEgKERSTSkgRm91 ckNDCj4gPj4+IGNvZGUgdG8gZ2l2ZSBhIGNvbXBsZXRlIGRlc2NyaXB0aW9uIG9mIHRoZSBkYXRh IGNvbnRhaW5lZCBpbiBhIGJ1ZmZlci4KPiA+Pj4gCj4gPj4+IFRoZSBzYW1lIG1vZGlmaWVyIGRl ZmluaXRpb24gaXMgdXNlZCBpbiB0aGUgS2hyb25vcyBFR0wgZXh0ZW5zaW9uCj4gPj4+IEVHTF9F WFRfaW1hZ2VfZG1hX2J1Zl9pbXBvcnRfbW9kaWZpZXJzLCBhbmQgaXMgc3VwcG9ydGVkIGluIHRo ZQo+ID4+PiBXYXlsYW5kIGxpbnV4LWRtYWJ1ZiBwcm90b2NvbC4KPiA+Pj4gCj4gPj4+IAo+ID4+ PiBUaGlzIGJ1ZmZlciBpbmZvcm1hdGlvbiBjb3VsZCBvZiBjb3Vyc2UgYmUgZGVzY3JpYmVkIGlu IHRoZQo+ID4+PiB2ZW5kb3Itc3BlY2lmaWMgcGFydCBvZiBWNEwyX1BJWF9GTVRfKiwgYnV0IHRo aXMgd291bGQgZHVwbGljYXRlIHRoZQo+ID4+PiBpbmZvcm1hdGlvbiBhbHJlYWR5IGRlZmluZWQg aW4gZHJtX2ZvdXJjYy5oLiBBZGRpdGlvbmFsbHksIHRoZXJlCj4gPj4+IHdvdWxkIGJlIHF1aXRl IGEgZm9ybWF0IGV4cGxvc2lvbiB3aGVyZSBhIGRldmljZSBzdXBwb3J0cyBhIGRvemVuIG9yCj4g Pj4+IG1vcmUgZm9ybWF0cywgYWxsIG9mIHdoaWNoIGNhbiB1c2Ugb25lIG9yIG1vcmUgZGlmZmVy ZW50Cj4gPj4+IGxheW91dHMvY29tcHJlc3Npb24gc2NoZW1lcy4KPiA+Pj4gCj4gPj4+IFNvLCBJ J20gd29uZGVyaW5nIGlmIGFueW9uZSBoYXMgdmlld3Mgb24gaG93L3doZXRoZXIgdGhpcyBjb3Vs ZCBiZQo+ID4+PiBpbmNvcnBvcmF0ZWQ/Cj4gPj4+IAo+ID4+PiBJIHNwb2tlIGJyaWVmbHkgYWJv dXQgdGhpcyB0byBMYXVyZW50IGF0IExQQyBsYXN0IHllYXIsIGFuZCBoZQo+ID4+PiBzdWdnZXN0 ZWQgdjRsMl9jb250cm9sIGFzIG9uZSBhcHByb2FjaC4KPiA+Pj4gCj4gPj4+IEkgYWxzbyB3b25k ZXJlZCBpZiBjb3VsZCBiZSBhZGRlZCBpbiB2NGwyX3BpeF9mb3JtYXRfbXBsYW5lIC0gbG9va3MK PiA+Pj4gbGlrZSB0aGVyZSdzIDggYnl0ZXMgbGVmdCBiZWZvcmUgaXQgZXhjZWVkcyB0aGUgMjAw IGJ5dGVzLCBvciBjb3VsZCBnbwo+ID4+PiBpbiB0aGUgcmVzZXJ2ZWQgcG9ydGlvbiBvZiB2NGwy X3BsYW5lX3BpeF9mb3JtYXQuCgpXZSdyZSBjb25zaWRlcmluZyByZXdvcmtpbmcgdGhlIGZvcm1h dCBpb2N0bHMgYXQgc29tZSBwb2ludC4gV2UgZG9uJ3QgCm5lY2Vzc2FyaWx5IG5lZWQgdG8gd2Fp dCB1bnRpbCB0aGVuIHRvIGltcGxlbWVudCBzdXBwb3J0IGZvciBtb2RpZmllcnMsIGJ1dCB3ZSAK d2lsbCBoYXZlIGFuIG9wcG9ydHVuaXR5IHRvIGludGVncmF0ZSB0aGVtIHdpdGggZm9ybWF0cyBh dCB0aGF0IHBvaW50LgoKPiA+Pj4gVGhhbmtzIGZvciBhbnkgdGhvdWdodHMsCj4gPj4gCj4gPj4g T25lIHByb2JsZW0gaXMgdGhhdCB0aGUgbW9kaWZlcnMgc29tZXRpbWVzIHJlZmVyZW5jZSB0aGUg RFJNIGZvdXJjYwo+ID4+IGNvZGVzLiB2NGwgaGFzIGEgZGlmZmVyZW50IChhbmQgaW5jb21wYXRp YmxlIHNldCkgb2YgZm91cmNjIGNvZGVzLAo+ID4+IHdoZXJlYXMgYWxsIHRoZSBwcm90b2NvbHMg YW5kIHNwZWNzICh5b3UgY2FuIGFkZCBEUkkzLjEgZm9yIFhvcmcgdG8KPiA+PiB0aGF0IGxpc3Qg YnR3KSB1c2UgYm90aCBkcm0gZm91cmNjIGFuZCBkcm0gbW9kaWZpZXJzLgo+ID4+IAo+ID4+IFRo aXMgbWlnaHQgb3IgbWlnaHQgbm90IG1ha2UgdGhpcyBwcm9wb3NhbCB1bndvcmthYmxlLCBidXQg aXQncwo+ID4+IHNvbWV0aGluZyBJJ2QgYXQgbGVhc3QgcmV2aWV3IGNhcmVmdWxseS4KPiA+PiAK PiA+PiBPdGhlcndpc2UgSSB0aGluayBpdCdkIGJlIGdyZWF0IGlmIHdlIGNvdWxkIGhhdmUgb25l IG5hbWVzcGFjZSBmb3IgYWxsCj4gPj4gbW9kaWZpZXJzLCB0aGF0J3MgcHJldHR5IG11Y2ggd2h5 IHdlIGhhdmUgdGhlbS4gUGxlYXNlIGFsc28gbm90ZSB0aGF0Cj4gPj4gZm9yIGRybV9mb3VyY2Mu aCB3ZSBkb24ndCByZXF1aXJlIGFuIGluLWtlcm5lbCB1c2VyIGZvciBhIG5ldyBtb2RpZmllcgo+ ID4+IHNpbmNlIGEgYnVuY2ggb2YgdGhlbSBtaWdodCBuZWVkIHRvIGJlIGFsbG9jYXRlZCBqdXN0 IGZvcgo+ID4+IHVzZXJzcGFjZS10by11c2Vyc3BhY2UgYnVmZmVyIHNoYXJpbmcgKGUuZy4gaW4g RUdML3ZrKS4gT25lIGV4YW1wbGUKPiA+PiBmb3IgdGhpcyB3b3VsZCBiZSBjb21wcmVzc2VkIHN1 cmZhY2VzIHdpdGggZmFzdC1jbGVhcmluZywgd2hpY2ggaXMKPiA+PiBwbGFubmVkIGZvciBpOTE1 IChidXQgY3VycmVudCBodyBjYW4ndCBzY2FuIGl0IG91dCkuIEFuZCB3ZSByZWFsbHkKPiA+PiB3 YW50IHRvIGhhdmUgb25lIG5hbWVzcGFjZSBmb3IgZXZlcnl0aGluZy4KPiA+Cj4gPiBXaG8gc2V0 cyB0aGVzZSBtb2RpZmllcnM/IEtlcm5lbCBvciB1c2Vyc3BhY2U/IE9yIGNhbiBpdCBiZSBzZXQg YnkgYm90aD8KPiA+IEkgYXNzdW1lIGFueSB1c2Vyc3BhY2UgY29kZSB0aGF0IHNldHMvcmVhZHMg dGhpcyBpcyBjb2RlIHNwZWNpZmljIGZvciB0aGF0Cj4gPiBoYXJkd2FyZT8KPiAKPiBJIHRoaW5r IG5vcm1hbGx5IHRoZSBtb2RpZmllciB3b3VsZCBiZSBzZXQgYnkgdXNlcnNwYWNlLiBIb3dldmVy IGl0Cj4gbWlnaHQgbm90IG5lY2Vzc2FyaWx5IGJlIGRldmljZS1zcGVjaWZpYyBjb2RlLiBJbiBE Uk0gdGhlIGludGVudGlvbiBpcwo+IGZvciB1c2Vyc3BhY2UgdG8gcXVlcnkgdGhlIHNldCBvZiBt b2RpZmllcnMgd2hpY2ggYXJlIHN1cHBvcnRlZCwgYW5kCj4gdGhlbiB1c2UgdGhlbSB3aXRob3V0 IG5lY2Vzc2FyaWx5IGtub3dpbmcgZXhhY3RseSB3aGF0IHRoZXkgbWVhbgo+IChpbnNvZmFyIGFz IHRoYXQgaXMgcG9zc2libGUpLgo+IAo+IGUuZy4gaWYgSSBoYXZlIHR3byBkZXZpY2VzIHdoaWNo IHN1cHBvcnQgTU9ESUZJRVJfRk9PLCBJIGNvdWxkIGF0dGVtcHQKPiB0byBzaGFyZSBhIGJ1ZmZl ciBiZXR3ZWVuIHRoZW0gd2hpY2ggdXNlcyBNT0RJRklFUl9GT08gd2l0aG91dAo+IG5lY2Vzc2Fy aWx5IGtub3dpbmcgZXhhY3RseSB3aGF0IGl0IGlzL2RvZXMuCgpVc2Vyc3BhY2UgY291bGQgY2Vy dGFpbmx5IHNldCBtb2RpZmllcnMgYmxpbmRseSwgYnV0IHRoZSBwb2ludCBvZiBtb2RpZmllcnMg aXMgCnRvIGdlbmVyYXRlIHNpZGUgZWZmZWN0cyBiZW5lZml0aWFsIHRvIHRoZSB1c2UgY2FzZSBh dCBoYW5kIChmb3IgaW5zdGFuY2UgYnkgCm9wdGltaXppbmcgdGhlIG1lbW9yeSBhY2Nlc3MgcGF0 dGVybikuIFRvIHVzZSB0aGVtIG1lYW5pbmdmdWxseSB1c2Vyc3BhY2UgCndvdWxkIG5lZWQgdG8g aGF2ZSBhdCBsZWFzdCBhbiBpZGVhIG9mIHRoZSBzaWRlIGVmZmVjdHMgdGhleSBnZW5lcmF0ZS4K Cj4gPiBJIHRoaW5rIExhdXJlbnQncyBzdWdnZXN0aW9uIG9mIHVzaW5nIGEgNjQgYml0IFY0TDIg Y29udHJvbCBmb3IgdGhpcyBtYWtlcwo+ID4gdGhlIG1vc3Qgc2Vuc2UuCj4gPgo+ID4gRXNwZWNp YWxseSBpZiB5b3UgY2FuIGFzc3VtZSB0aGF0IHdob2V2ZXIgc2V0cyB0aGlzIGtub3dzIHRoZSBo YXJkd2FyZS4KPiA+Cj4gPiBJIHRoaW5rIHRoaXMgb25seSBtYWtlcyBzZW5zZSBpZiB5b3UgcGFz cyBidWZmZXJzIGZyb20gb25lIEhXIGRldmljZSB0bwo+ID4gYW5vdGhlci4KPiA+Cj4gPiBCZWNh dXNlIHlvdSBjYW5ub3QgZXhwZWN0IGdlbmVyaWMgdmlkZW8gY2FwdHVyZSBjb2RlIHRvIGJlIGFi bGUgdG8KPiA+IGludGVycHJldCBhbGwgdGhlIHppbGxpb24gZGlmZmVyZW50IGNvbWJpbmF0aW9u cyBvZiBtb2RpZmllcnMuCj4gCj4gSSBkb24ndCBxdWl0ZSBmb2xsb3cgdGhpcyBsYXN0IGJpdC4g VGhlIGNvbnRyb2wgY291bGQgcmVwb3J0IHRoZSBzZXQKPiBvZiBzdXBwb3J0ZWQgbW9kaWZpZXJz Lgo+IAo+IEhvd2V2ZXIsIGluIERSTSB0aGUgQVBJIGxldHMgeW91IGdldCB0aGUgc3VwcG9ydGVk IGZvcm1hdHMgZm9yIGVhY2gKPiBtb2RpZmllciBhcy13ZWxsLWFzIHRoZSBtb2RpZmllciBsaXN0 IGl0c2VsZi4gSSdtIG5vdCBzdXJlIGhvdyBleGFjdGx5Cj4gdG8gcHJvdmlkZSB0aGF0IGluIGEg Y29udHJvbC4KCi0tIApSZWdhcmRzLAoKTGF1cmVudCBQaW5jaGFydAoKX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KZHJpLWRldmVsIG1haWxpbmcgbGlzdApk cmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Au b3JnL21haWxtYW4vbGlzdGluZm8vZHJpLWRldmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from galahad.ideasonboard.com ([185.26.127.97]:38503 "EHLO galahad.ideasonboard.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751086AbdHaO2h (ORCPT ); Thu, 31 Aug 2017 10:28:37 -0400 From: Laurent Pinchart To: Brian Starkey Cc: Hans Verkuil , Daniel Vetter , "linux-media@vger.kernel.org" , jonathan.chai@arm.com, dri-devel Subject: Re: DRM Format Modifiers in v4l2 Date: Thu, 31 Aug 2017 17:28:54 +0300 Message-ID: <4559442.sz5HF0f0o4@avalon> In-Reply-To: <20170824111430.GB25711@e107564-lin.cambridge.arm.com> References: <20170821155203.GB38943@e107564-lin.cambridge.arm.com> <47128f36-2990-bd45-ead9-06a31ed8cde0@xs4all.nl> <20170824111430.GB25711@e107564-lin.cambridge.arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Sender: linux-media-owner@vger.kernel.org List-ID: Hi Brian, On Thursday, 24 August 2017 14:14:31 EEST Brian Starkey wrote: > On Mon, Aug 21, 2017 at 06:36:29PM +0200, Hans Verkuil wrote: > > On 08/21/2017 06:01 PM, Daniel Vetter wrote: > >> On Mon, Aug 21, 2017 at 5:52 PM, Brian Starkey wrote: > >>> Hi all, > >>> > >>> I couldn't find this topic talked about elsewhere, but apologies if > >>> it's a duplicate - I'll be glad to be steered in the direction of a > >>> thread. > >>> > >>> We'd like to support DRM format modifiers in v4l2 in order to share > >>> the description of different (mostly proprietary) buffer formats > >>> between e.g. a v4l2 device and a DRM device. > >>> > >>> DRM format modifiers are defined in include/uapi/drm/drm_fourcc.h and > >>> are a vendor-namespaced 64-bit value used to describe various > >>> vendor-specific buffer layouts. They are combined with a (DRM) FourCC > >>> code to give a complete description of the data contained in a buffer. > >>> > >>> The same modifier definition is used in the Khronos EGL extension > >>> EGL_EXT_image_dma_buf_import_modifiers, and is supported in the > >>> Wayland linux-dmabuf protocol. > >>> > >>> > >>> This buffer information could of course be described in the > >>> vendor-specific part of V4L2_PIX_FMT_*, but this would duplicate the > >>> information already defined in drm_fourcc.h. Additionally, there > >>> would be quite a format explosion where a device supports a dozen or > >>> more formats, all of which can use one or more different > >>> layouts/compression schemes. > >>> > >>> So, I'm wondering if anyone has views on how/whether this could be > >>> incorporated? > >>> > >>> I spoke briefly about this to Laurent at LPC last year, and he > >>> suggested v4l2_control as one approach. > >>> > >>> I also wondered if could be added in v4l2_pix_format_mplane - looks > >>> like there's 8 bytes left before it exceeds the 200 bytes, or could go > >>> in the reserved portion of v4l2_plane_pix_format. We're considering reworking the format ioctls at some point. We don't necessarily need to wait until then to implement support for modifiers, but we will have an opportunity to integrate them with formats at that point. > >>> Thanks for any thoughts, > >> > >> One problem is that the modifers sometimes reference the DRM fourcc > >> codes. v4l has a different (and incompatible set) of fourcc codes, > >> whereas all the protocols and specs (you can add DRI3.1 for Xorg to > >> that list btw) use both drm fourcc and drm modifiers. > >> > >> This might or might not make this proposal unworkable, but it's > >> something I'd at least review carefully. > >> > >> Otherwise I think it'd be great if we could have one namespace for all > >> modifiers, that's pretty much why we have them. Please also note that > >> for drm_fourcc.h we don't require an in-kernel user for a new modifier > >> since a bunch of them might need to be allocated just for > >> userspace-to-userspace buffer sharing (e.g. in EGL/vk). One example > >> for this would be compressed surfaces with fast-clearing, which is > >> planned for i915 (but current hw can't scan it out). And we really > >> want to have one namespace for everything. > > > > Who sets these modifiers? Kernel or userspace? Or can it be set by both? > > I assume any userspace code that sets/reads this is code specific for that > > hardware? > > I think normally the modifier would be set by userspace. However it > might not necessarily be device-specific code. In DRM the intention is > for userspace to query the set of modifiers which are supported, and > then use them without necessarily knowing exactly what they mean > (insofar as that is possible). > > e.g. if I have two devices which support MODIFIER_FOO, I could attempt > to share a buffer between them which uses MODIFIER_FOO without > necessarily knowing exactly what it is/does. Userspace could certainly set modifiers blindly, but the point of modifiers is to generate side effects benefitial to the use case at hand (for instance by optimizing the memory access pattern). To use them meaningfully userspace would need to have at least an idea of the side effects they generate. > > I think Laurent's suggestion of using a 64 bit V4L2 control for this makes > > the most sense. > > > > Especially if you can assume that whoever sets this knows the hardware. > > > > I think this only makes sense if you pass buffers from one HW device to > > another. > > > > Because you cannot expect generic video capture code to be able to > > interpret all the zillion different combinations of modifiers. > > I don't quite follow this last bit. The control could report the set > of supported modifiers. > > However, in DRM the API lets you get the supported formats for each > modifier as-well-as the modifier list itself. I'm not sure how exactly > to provide that in a control. -- Regards, Laurent Pinchart