multi-planar tiled fourcc's in mesa and drm

multi-planar tiled fourcc's in mesa and drm

[Rob Clark]

So, lucky me, I have a scenario where I get to deal with NV12MT.  Hurray!

I know there has been some reluctance in the past to combine tiling
and color format, since in theory that could lead to a combinatorial
explosion in formats.  And, as long as the buffer usage is entirely
within a single driver, you can approximately hide tiling (or
compressed, etc) permutations of a color format.  On the other hand,
there is already some precedence for fourcc or format values to
represent tiled formats at the interface level (in kernel, v4l, and in
userspace, and gstreamer and openmax).

But in this scenario, sharing buffer between other devices (video
decoder/encoder) and drm/kms (msm) and mesa (freedreno) via
EGL_EXT_image_dma_buf_import[1], I sort of don't really have any other
way to pass around tiling flags.  So I would propose adding custom
fourcc's only in the more limited cases where formats are exchanged
between devices.  This should avoid an explosion of color_format *
tiling_format.

For the kms part, it would mean merging a small patch to allow addfb2
for NV12MT[2].

For the mesa part, it looks like there is a bit of work needed to
teach egl about multi-planar buffers, buffers where offset[n] != 0,
etc.  I'll start with patches to teach egl how to import plain NV12
buffers.  But once that is done, for it to be much use to me I'll need
NV12MT, which means adding a new gallium format and
__DRI_IMAGE_FOURCC_NV12MT.

Also, I'm still a bit undecided on how to represent multi-planar
formats (ie. single pipe_resource encapsulating each of the planes?
or pipe_resource per plane but teach pipe_sampler_view about textures
which have multiple pipe_resource's, one for per plane).

Anyways, I'll start working on the mesa egl bits next week.  First
step is just add an 'offset' parameter to 'struct winsys_handle',
which should hopefully be non-controversial.  After that, I need to
decide how to handle multi-planar, and I think that hinges on how
folks want to handle multi-planar in gallium.  Ie. if one
pipe_resource per plane, then winsys_handle doesn't need any further
change (but we need changes elsewhere), otherwise winsys_handle needs
to have an array of handles.

Anyways, I'd appreciate feedback.

BR,
-R

[1] https://www.khronos.org/registry/egl/extensions/EXT/EGL_EXT_image_dma_buf_import.txt
[2] http://lists.freedesktop.org/archives/dri-devel/2014-July/064828.html

Re: multi-planar tiled fourcc's in mesa and drm

[Rob Clark]

On Fri, Sep 19, 2014 at 3:16 PM, Rob Clark <robdclark@gmail.com> wrote:
> For the mesa part, it looks like there is a bit of work needed to
> teach egl about multi-planar buffers, buffers where offset[n] != 0,
> etc.  I'll start with patches to teach egl how to import plain NV12
> buffers.  But once that is done, for it to be much use to me I'll need
> NV12MT, which means adding a new gallium format and
> __DRI_IMAGE_FOURCC_NV12MT.
>
> Also, I'm still a bit undecided on how to represent multi-planar
> formats (ie. single pipe_resource encapsulating each of the planes?
> or pipe_resource per plane but teach pipe_sampler_view about textures
> which have multiple pipe_resource's, one for per plane).


So, on the mesa end of things, pipe_video_buffer looks like it may be
a better fit for an imported multi-planar format external eglimage
(since at least on some hw sampling a YUV buffer as a texture would
take multiple texture sampler slots), other than the fact that we
wouldn't have any codec in this case..  but does mesa state tracker
understand how to use a pipe_video_buffer as a sampler in a shader
somehow?  Right now I can only see references to pipe_video_buffer
from gallium video stuff.  I'd really prefer not to have to introduce
an extra YUV->RGB blit just to get the video frame into a form that
can be used from GL..

How does the connection between eglImage and omx state tracker work?
I'm probably getting at least a bit confused by the cpp macro hell in
bellagio headers..

BR,
-R

Re: multi-planar tiled fourcc's in mesa and drm

[Ilia Mirkin]

On Sat, Sep 20, 2014 at 11:34 AM, Rob Clark <robdclark@gmail.com> wrote:
> On Fri, Sep 19, 2014 at 3:16 PM, Rob Clark <robdclark@gmail.com> wrote:
>> For the mesa part, it looks like there is a bit of work needed to
>> teach egl about multi-planar buffers, buffers where offset[n] != 0,
>> etc.  I'll start with patches to teach egl how to import plain NV12
>> buffers.  But once that is done, for it to be much use to me I'll need
>> NV12MT, which means adding a new gallium format and
>> __DRI_IMAGE_FOURCC_NV12MT.
>>
>> Also, I'm still a bit undecided on how to represent multi-planar
>> formats (ie. single pipe_resource encapsulating each of the planes?
>> or pipe_resource per plane but teach pipe_sampler_view about textures
>> which have multiple pipe_resource's, one for per plane).
>
>
> So, on the mesa end of things, pipe_video_buffer looks like it may be
> a better fit for an imported multi-planar format external eglimage
> (since at least on some hw sampling a YUV buffer as a texture would
> take multiple texture sampler slots), other than the fact that we
> wouldn't have any codec in this case..  but does mesa state tracker
> understand how to use a pipe_video_buffer as a sampler in a shader
> somehow?  Right now I can only see references to pipe_video_buffer
> from gallium video stuff.  I'd really prefer not to have to introduce
> an extra YUV->RGB blit just to get the video frame into a form that
> can be used from GL..

You may be interested in vl_compositor.c which details with these
pipe_video_buffer's. You can use the pipe_video_buffer's
->get_sampler_view_components to get the individual YUV bits (or
whatever the format).

>
> How does the connection between eglImage and omx state tracker work?
> I'm probably getting at least a bit confused by the cpp macro hell in
> bellagio headers..
>
> BR,
> -R
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: multi-planar tiled fourcc's in mesa and drm

[Rob Clark]

On Sat, Sep 20, 2014 at 11:40 AM, Ilia Mirkin <imirkin@alum.mit.edu> wrote:
> On Sat, Sep 20, 2014 at 11:34 AM, Rob Clark <robdclark@gmail.com> wrote:
>> On Fri, Sep 19, 2014 at 3:16 PM, Rob Clark <robdclark@gmail.com> wrote:
>>> For the mesa part, it looks like there is a bit of work needed to
>>> teach egl about multi-planar buffers, buffers where offset[n] != 0,
>>> etc.  I'll start with patches to teach egl how to import plain NV12
>>> buffers.  But once that is done, for it to be much use to me I'll need
>>> NV12MT, which means adding a new gallium format and
>>> __DRI_IMAGE_FOURCC_NV12MT.
>>>
>>> Also, I'm still a bit undecided on how to represent multi-planar
>>> formats (ie. single pipe_resource encapsulating each of the planes?
>>> or pipe_resource per plane but teach pipe_sampler_view about textures
>>> which have multiple pipe_resource's, one for per plane).
>>
>>
>> So, on the mesa end of things, pipe_video_buffer looks like it may be
>> a better fit for an imported multi-planar format external eglimage
>> (since at least on some hw sampling a YUV buffer as a texture would
>> take multiple texture sampler slots), other than the fact that we
>> wouldn't have any codec in this case..  but does mesa state tracker
>> understand how to use a pipe_video_buffer as a sampler in a shader
>> somehow?  Right now I can only see references to pipe_video_buffer
>> from gallium video stuff.  I'd really prefer not to have to introduce
>> an extra YUV->RGB blit just to get the video frame into a form that
>> can be used from GL..
>
> You may be interested in vl_compositor.c which details with these
> pipe_video_buffer's. You can use the pipe_video_buffer's
> ->get_sampler_view_components to get the individual YUV bits (or
> whatever the format).

hmm, I suppose it is a possibility to use vl_compositor from gallium
egl bits to blit the YUV into an RGB shadow buffer, and then use that
as the eglImage.  Not exactly awesome from a memory bandwidth
perspective, and I'd need to think a bit about how the bookkeeping
would work.  But afaiu this would be allowed for external eglImages,
so I suppose it might be the best approach just to get something
working.

>>
>> How does the connection between eglImage and omx state tracker work?
>> I'm probably getting at least a bit confused by the cpp macro hell in
>> bellagio headers..

fwiw, afaict so far, the mesa omx does not support eglImage..

>> BR,
>> -R
>> _______________________________________________
>> dri-devel mailing list
>> dri-devel@lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/dri-devel