[RFC]: shmem fd for non-DMA buffer sharing cross drivers

[RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Hsia-Jun Li]

Hello

I would like to introduce a usage of SHMEM slimier to DMA-buf, the major 
purpose of that is sharing metadata or just a pure container for cross 
drivers.

We need to exchange some sort of metadata between drivers, likes dynamic 
HDR data between video4linux2 and DRM. Or the graphics frame buffer is 
too complex to be described with plain plane's DMA-buf fd.
An issue between DRM and V4L2 is that DRM could only support 4 planes 
while it is 8 for V4L2. It would be pretty hard for DRM to expend its 
interface to support that 4 more planes which would lead to revision of 
many standard likes Vulkan, EGL.

Also, there is no reason to consume a device's memory for the content 
that device can't read it, or wasting an entry of IOMMU for such data.
Usually, such a metadata would be the value should be written to a 
hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.

Still, I have some problems with SHMEM:
1. I don't want thhe userspace modify the context of the SHMEM allocated 
by the kernel, is there a way to do so?
2. Should I create a helper function for installing the SHMEM file as a fd?

-- 
Hsia-Jun(Randy) Li

[RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Hsia-Jun Li]

Hello

I would like to introduce a usage of SHMEM slimier to DMA-buf, the major 
purpose of that is sharing metadata or just a pure container for cross 
drivers.

We need to exchange some sort of metadata between drivers, likes dynamic 
HDR data between video4linux2 and DRM. Or the graphics frame buffer is 
too complex to be described with plain plane's DMA-buf fd.
An issue between DRM and V4L2 is that DRM could only support 4 planes 
while it is 8 for V4L2. It would be pretty hard for DRM to expend its 
interface to support that 4 more planes which would lead to revision of 
many standard likes Vulkan, EGL.

Also, there is no reason to consume a device's memory for the content 
that device can't read it, or wasting an entry of IOMMU for such data.
Usually, such a metadata would be the value should be written to a 
hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.

Still, I have some problems with SHMEM:
1. I don't want thhe userspace modify the context of the SHMEM allocated 
by the kernel, is there a way to do so?
2. Should I create a helper function for installing the SHMEM file as a fd?

-- 
Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Nicolas Dufresne]

Hi,

Le mardi 22 août 2023 à 19:14 +0800, Hsia-Jun Li a écrit :
> Hello
> 
> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major 
> purpose of that is sharing metadata or just a pure container for cross 
> drivers.
> 
> We need to exchange some sort of metadata between drivers, likes dynamic 
> HDR data between video4linux2 and DRM. Or the graphics frame buffer is 
> too complex to be described with plain plane's DMA-buf fd.
> An issue between DRM and V4L2 is that DRM could only support 4 planes 
> while it is 8 for V4L2. It would be pretty hard for DRM to expend its 
> interface to support that 4 more planes which would lead to revision of 
> many standard likes Vulkan, EGL.
> 
> Also, there is no reason to consume a device's memory for the content 
> that device can't read it, or wasting an entry of IOMMU for such data.
> Usually, such a metadata would be the value should be written to a 
> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
> 
> Still, I have some problems with SHMEM:
> 1. I don't want thhe userspace modify the context of the SHMEM allocated 
> by the kernel, is there a way to do so?
> 2. Should I create a helper function for installing the SHMEM file as a fd?

Please have a look at memfd and the seal feature, it does cover the reason why
unsealed shared memory require full trust. For controls, the SEAL_WRITE is even
needed, as with appropriate timing, a malicous process can modify the data in-
between validation and allocation, causing possible memory overflow.

https://man7.org/linux/man-pages/man2/memfd_create.2.html
File sealing
       In the absence of file sealing, processes that communicate via
       shared memory must either trust each other, or take measures to
       deal with the possibility that an untrusted peer may manipulate
       the shared memory region in problematic ways.  For example, an
       untrusted peer might modify the contents of the shared memory at
       any time, or shrink the shared memory region.  The former
       possibility leaves the local process vulnerable to time-of-check-
       to-time-of-use race conditions (typically dealt with by copying
       data from the shared memory region before checking and using it).
       The latter possibility leaves the local process vulnerable to
       SIGBUS signals when an attempt is made to access a now-
       nonexistent location in the shared memory region.  (Dealing with
       this possibility necessitates the use of a handler for the SIGBUS
       signal.)

       Dealing with untrusted peers imposes extra complexity on code
       that employs shared memory.  Memory sealing enables that extra
       complexity to be eliminated, by allowing a process to operate
       secure in the knowledge that its peer can't modify the shared
       memory in an undesired fashion.

       [...]

regards,
Nicolas

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Nicolas Dufresne]

Hi,

Le mardi 22 août 2023 à 19:14 +0800, Hsia-Jun Li a écrit :
> Hello
> 
> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major 
> purpose of that is sharing metadata or just a pure container for cross 
> drivers.
> 
> We need to exchange some sort of metadata between drivers, likes dynamic 
> HDR data between video4linux2 and DRM. Or the graphics frame buffer is 
> too complex to be described with plain plane's DMA-buf fd.
> An issue between DRM and V4L2 is that DRM could only support 4 planes 
> while it is 8 for V4L2. It would be pretty hard for DRM to expend its 
> interface to support that 4 more planes which would lead to revision of 
> many standard likes Vulkan, EGL.
> 
> Also, there is no reason to consume a device's memory for the content 
> that device can't read it, or wasting an entry of IOMMU for such data.
> Usually, such a metadata would be the value should be written to a 
> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
> 
> Still, I have some problems with SHMEM:
> 1. I don't want thhe userspace modify the context of the SHMEM allocated 
> by the kernel, is there a way to do so?
> 2. Should I create a helper function for installing the SHMEM file as a fd?

Please have a look at memfd and the seal feature, it does cover the reason why
unsealed shared memory require full trust. For controls, the SEAL_WRITE is even
needed, as with appropriate timing, a malicous process can modify the data in-
between validation and allocation, causing possible memory overflow.

https://man7.org/linux/man-pages/man2/memfd_create.2.html
File sealing
       In the absence of file sealing, processes that communicate via
       shared memory must either trust each other, or take measures to
       deal with the possibility that an untrusted peer may manipulate
       the shared memory region in problematic ways.  For example, an
       untrusted peer might modify the contents of the shared memory at
       any time, or shrink the shared memory region.  The former
       possibility leaves the local process vulnerable to time-of-check-
       to-time-of-use race conditions (typically dealt with by copying
       data from the shared memory region before checking and using it).
       The latter possibility leaves the local process vulnerable to
       SIGBUS signals when an attempt is made to access a now-
       nonexistent location in the shared memory region.  (Dealing with
       this possibility necessitates the use of a handler for the SIGBUS
       signal.)

       Dealing with untrusted peers imposes extra complexity on code
       that employs shared memory.  Memory sealing enables that extra
       complexity to be eliminated, by allowing a process to operate
       secure in the knowledge that its peer can't modify the shared
       memory in an undesired fashion.

       [...]

regards,
Nicolas

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Hsia-Jun Li]

On 8/23/23 03:55, Nicolas Dufresne wrote:
> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> Hi,
> 
> Le mardi 22 août 2023 à 19:14 +0800, Hsia-Jun Li a écrit :
>> Hello
>>
>> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
>> purpose of that is sharing metadata or just a pure container for cross
>> drivers.
>>
>> We need to exchange some sort of metadata between drivers, likes dynamic
>> HDR data between video4linux2 and DRM. Or the graphics frame buffer is
>> too complex to be described with plain plane's DMA-buf fd.
>> An issue between DRM and V4L2 is that DRM could only support 4 planes
>> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
>> interface to support that 4 more planes which would lead to revision of
>> many standard likes Vulkan, EGL.
>>
>> Also, there is no reason to consume a device's memory for the content
>> that device can't read it, or wasting an entry of IOMMU for such data.
>> Usually, such a metadata would be the value should be written to a
>> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
>>
>> Still, I have some problems with SHMEM:
>> 1. I don't want thhe userspace modify the context of the SHMEM allocated
>> by the kernel, is there a way to do so?
>> 2. Should I create a helper function for installing the SHMEM file as a fd?
> 
> Please have a look at memfd and the seal feature, it does cover the reason why

That is the implement I need, it would affact the userspace not the 
kernel space. Should I expand a kAPI for memfd or just take the 
implement for the SHMEM?
This interfaces need to offer three things:
1. a fd for userspace to exchange between drivers
2. a kernel virtual address for accessing
3. userspace SEAL

Meanwhile, I am thinking whether we should offer a generic context 
header for such usage. Or we need another fields in a driver to describe it.
struct shmem_generic_container {
	u64 format; /* use DRM modifier vendor bits but */
	u32 size; /* size of the payload */
	u8 payload[];
};
/* format linear for nesting dolls context */
struct shmem_nesting_container {
	u32 num;
	u64 formats[num];
	u32 sizes[num];
	u32 offsets[num]; /* offset from the payload below */
	u8 payload[];
};
> unsealed shared memory require full trust. For controls, the SEAL_WRITE is even
> needed, as with appropriate timing, a malicous process can modify the data in-
> between validation and allocation, causing possible memory overflow.
> 
> https://man7.org/linux/man-pages/man2/memfd_create.2.html
> File sealing
>         In the absence of file sealing, processes that communicate via
>         shared memory must either trust each other, or take measures to
>         deal with the possibility that an untrusted peer may manipulate
>         the shared memory region in problematic ways.  For example, an
>         untrusted peer might modify the contents of the shared memory at
>         any time, or shrink the shared memory region.  The former
>         possibility leaves the local process vulnerable to time-of-check-
>         to-time-of-use race conditions (typically dealt with by copying
>         data from the shared memory region before checking and using it).
>         The latter possibility leaves the local process vulnerable to
>         SIGBUS signals when an attempt is made to access a now-
>         nonexistent location in the shared memory region.  (Dealing with
>         this possibility necessitates the use of a handler for the SIGBUS
>         signal.)
> 
>         Dealing with untrusted peers imposes extra complexity on code
>         that employs shared memory.  Memory sealing enables that extra
>         complexity to be eliminated, by allowing a process to operate
>         secure in the knowledge that its peer can't modify the shared
>         memory in an undesired fashion.
> 
>         [...]
> 
> regards,
> Nicolas

-- 
Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Hsia-Jun Li]

On 8/23/23 03:55, Nicolas Dufresne wrote:
> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> Hi,
> 
> Le mardi 22 août 2023 à 19:14 +0800, Hsia-Jun Li a écrit :
>> Hello
>>
>> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
>> purpose of that is sharing metadata or just a pure container for cross
>> drivers.
>>
>> We need to exchange some sort of metadata between drivers, likes dynamic
>> HDR data between video4linux2 and DRM. Or the graphics frame buffer is
>> too complex to be described with plain plane's DMA-buf fd.
>> An issue between DRM and V4L2 is that DRM could only support 4 planes
>> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
>> interface to support that 4 more planes which would lead to revision of
>> many standard likes Vulkan, EGL.
>>
>> Also, there is no reason to consume a device's memory for the content
>> that device can't read it, or wasting an entry of IOMMU for such data.
>> Usually, such a metadata would be the value should be written to a
>> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
>>
>> Still, I have some problems with SHMEM:
>> 1. I don't want thhe userspace modify the context of the SHMEM allocated
>> by the kernel, is there a way to do so?
>> 2. Should I create a helper function for installing the SHMEM file as a fd?
> 
> Please have a look at memfd and the seal feature, it does cover the reason why

That is the implement I need, it would affact the userspace not the 
kernel space. Should I expand a kAPI for memfd or just take the 
implement for the SHMEM?
This interfaces need to offer three things:
1. a fd for userspace to exchange between drivers
2. a kernel virtual address for accessing
3. userspace SEAL

Meanwhile, I am thinking whether we should offer a generic context 
header for such usage. Or we need another fields in a driver to describe it.
struct shmem_generic_container {
	u64 format; /* use DRM modifier vendor bits but */
	u32 size; /* size of the payload */
	u8 payload[];
};
/* format linear for nesting dolls context */
struct shmem_nesting_container {
	u32 num;
	u64 formats[num];
	u32 sizes[num];
	u32 offsets[num]; /* offset from the payload below */
	u8 payload[];
};
> unsealed shared memory require full trust. For controls, the SEAL_WRITE is even
> needed, as with appropriate timing, a malicous process can modify the data in-
> between validation and allocation, causing possible memory overflow.
> 
> https://man7.org/linux/man-pages/man2/memfd_create.2.html
> File sealing
>         In the absence of file sealing, processes that communicate via
>         shared memory must either trust each other, or take measures to
>         deal with the possibility that an untrusted peer may manipulate
>         the shared memory region in problematic ways.  For example, an
>         untrusted peer might modify the contents of the shared memory at
>         any time, or shrink the shared memory region.  The former
>         possibility leaves the local process vulnerable to time-of-check-
>         to-time-of-use race conditions (typically dealt with by copying
>         data from the shared memory region before checking and using it).
>         The latter possibility leaves the local process vulnerable to
>         SIGBUS signals when an attempt is made to access a now-
>         nonexistent location in the shared memory region.  (Dealing with
>         this possibility necessitates the use of a handler for the SIGBUS
>         signal.)
> 
>         Dealing with untrusted peers imposes extra complexity on code
>         that employs shared memory.  Memory sealing enables that extra
>         complexity to be eliminated, by allowing a process to operate
>         secure in the knowledge that its peer can't modify the shared
>         memory in an undesired fashion.
> 
>         [...]
> 
> regards,
> Nicolas

-- 
Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Tomasz Figa]

Hi Hsia-Jun,

On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
>
> Hello
>
> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
> purpose of that is sharing metadata or just a pure container for cross
> drivers.
>
> We need to exchange some sort of metadata between drivers, likes dynamic
> HDR data between video4linux2 and DRM.

If the metadata isn't too big, would it be enough to just have the
kernel copy_from_user() to a kernel buffer in the ioctl code?

> Or the graphics frame buffer is
> too complex to be described with plain plane's DMA-buf fd.
> An issue between DRM and V4L2 is that DRM could only support 4 planes
> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
> interface to support that 4 more planes which would lead to revision of
> many standard likes Vulkan, EGL.

Could you explain how a shmem buffer could be used to support frame
buffers with more than 4 planes?

>
> Also, there is no reason to consume a device's memory for the content
> that device can't read it, or wasting an entry of IOMMU for such data.

That's right, but DMA-buf doesn't really imply any of those. DMA-buf
is just a kernel object with some backing memory. It's up to the
allocator to decide how the backing memory is allocated and up to the
importer on whether it would be mapped into an IOMMU.

> Usually, such a metadata would be the value should be written to a
> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
>
> Still, I have some problems with SHMEM:
> 1. I don't want thhe userspace modify the context of the SHMEM allocated
> by the kernel, is there a way to do so?

This is generally impossible without doing any of the two:
1) copying the contents to an internal buffer not accessible to the
userspace, OR
2) modifying any of the buffer mappings to read-only

2) can actually be more costly than 1) (depending on the architecture,
data size, etc.), so we shouldn't just discard the option of a simple
copy_from_user() in the ioctl.

> 2. Should I create a helper function for installing the SHMEM file as a fd?

We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
so maybe that would be enough?

[1] https://elixir.bootlin.com/linux/v6.5-rc7/source/drivers/dma-buf/udmabuf.c

Best,
Tomasz

>
> --
> Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Tomasz Figa]

Hi Hsia-Jun,

On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
>
> Hello
>
> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
> purpose of that is sharing metadata or just a pure container for cross
> drivers.
>
> We need to exchange some sort of metadata between drivers, likes dynamic
> HDR data between video4linux2 and DRM.

If the metadata isn't too big, would it be enough to just have the
kernel copy_from_user() to a kernel buffer in the ioctl code?

> Or the graphics frame buffer is
> too complex to be described with plain plane's DMA-buf fd.
> An issue between DRM and V4L2 is that DRM could only support 4 planes
> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
> interface to support that 4 more planes which would lead to revision of
> many standard likes Vulkan, EGL.

Could you explain how a shmem buffer could be used to support frame
buffers with more than 4 planes?

>
> Also, there is no reason to consume a device's memory for the content
> that device can't read it, or wasting an entry of IOMMU for such data.

That's right, but DMA-buf doesn't really imply any of those. DMA-buf
is just a kernel object with some backing memory. It's up to the
allocator to decide how the backing memory is allocated and up to the
importer on whether it would be mapped into an IOMMU.

> Usually, such a metadata would be the value should be written to a
> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
>
> Still, I have some problems with SHMEM:
> 1. I don't want thhe userspace modify the context of the SHMEM allocated
> by the kernel, is there a way to do so?

This is generally impossible without doing any of the two:
1) copying the contents to an internal buffer not accessible to the
userspace, OR
2) modifying any of the buffer mappings to read-only

2) can actually be more costly than 1) (depending on the architecture,
data size, etc.), so we shouldn't just discard the option of a simple
copy_from_user() in the ioctl.

> 2. Should I create a helper function for installing the SHMEM file as a fd?

We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
so maybe that would be enough?

[1] https://elixir.bootlin.com/linux/v6.5-rc7/source/drivers/dma-buf/udmabuf.c

Best,
Tomasz

>
> --
> Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Hsia-Jun Li]

On 8/23/23 12:46, Tomasz Figa wrote:
> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> Hi Hsia-Jun,
> 
> On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
>>
>> Hello
>>
>> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
>> purpose of that is sharing metadata or just a pure container for cross
>> drivers.
>>
>> We need to exchange some sort of metadata between drivers, likes dynamic
>> HDR data between video4linux2 and DRM.
> 
> If the metadata isn't too big, would it be enough to just have the
> kernel copy_from_user() to a kernel buffer in the ioctl code?
> 
>> Or the graphics frame buffer is
>> too complex to be described with plain plane's DMA-buf fd.
>> An issue between DRM and V4L2 is that DRM could only support 4 planes
>> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
>> interface to support that 4 more planes which would lead to revision of
>> many standard likes Vulkan, EGL.
> 
> Could you explain how a shmem buffer could be used to support frame
> buffers with more than 4 planes?
> If you are asking why we need this:
1. metadata likes dynamic HDR tone data
2. DRM also challenges with this problem, let me quote what sima said:
"another trick that we iirc used for afbc is that sometimes the planes 
have a fixed layout
like nv12
and so logically it's multiple planes, but you only need one plane slot 
to describe the buffer
since I think afbc had the "we need more than 4 planes" issue too"

Unfortunately, there are vendor pixel formats are not fixed layout.

3. Secure(REE, trusted video piepline) info.

For how to assign such metadata data.
In case with a drm fb_id, it is simple, we just add a drm plane property 
for it. The V4L2 interface is not flexible, we could only leave into 
CAPTURE request_fd as a control.
>>
>> Also, there is no reason to consume a device's memory for the content
>> that device can't read it, or wasting an entry of IOMMU for such data.
> 
> That's right, but DMA-buf doesn't really imply any of those. DMA-buf
> is just a kernel object with some backing memory. It's up to the
> allocator to decide how the backing memory is allocated and up to the
> importer on whether it would be mapped into an IOMMU.
> 
I just want to say it can't be allocated at the same place which was for 
those DMA bufs(graphics or compressed bitstream).
This also could be answer for your first question, if we place this kind 
of buffer in a plane for DMABUF(importing) in V4L2, V4L2 core would try 
to prepare it, which could map it into IOMMU.

>> Usually, such a metadata would be the value should be written to a
>> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
>>
>> Still, I have some problems with SHMEM:
>> 1. I don't want the userspace modify the context of the SHMEM allocated
>> by the kernel, is there a way to do so?
> 
> This is generally impossible without doing any of the two:
> 1) copying the contents to an internal buffer not accessible to the
> userspace, OR
> 2) modifying any of the buffer mappings to read-only
> 
> 2) can actually be more costly than 1) (depending on the architecture,
> data size, etc.), so we shouldn't just discard the option of a simple
> copy_from_user() in the ioctl.
> 
I don't want the userspace access it at all. So that won't be a problem.
>> 2. Should I create a helper function for installing the SHMEM file as a fd?
> 
> We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
> so maybe that would be enough?
> 
> [1] https://elixir.bootlin.com/linux/v6.5-rc7/source/drivers/dma-buf/udmabuf.c
> 
It is the kernel driver that allocate this buffer. For example, v4l2 
CAPTURE allocate a buffer for metadata when VIDIOC_REQBUFS.
Or GBM give you a fd which is assigned with a surface.

So we need a kernel interface.
> Best,
> Tomasz
> 
>>
>> --
>> Hsia-Jun(Randy) Li

-- 
Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Hsia-Jun Li]

On 8/23/23 12:46, Tomasz Figa wrote:
> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> Hi Hsia-Jun,
> 
> On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
>>
>> Hello
>>
>> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
>> purpose of that is sharing metadata or just a pure container for cross
>> drivers.
>>
>> We need to exchange some sort of metadata between drivers, likes dynamic
>> HDR data between video4linux2 and DRM.
> 
> If the metadata isn't too big, would it be enough to just have the
> kernel copy_from_user() to a kernel buffer in the ioctl code?
> 
>> Or the graphics frame buffer is
>> too complex to be described with plain plane's DMA-buf fd.
>> An issue between DRM and V4L2 is that DRM could only support 4 planes
>> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
>> interface to support that 4 more planes which would lead to revision of
>> many standard likes Vulkan, EGL.
> 
> Could you explain how a shmem buffer could be used to support frame
> buffers with more than 4 planes?
> If you are asking why we need this:
1. metadata likes dynamic HDR tone data
2. DRM also challenges with this problem, let me quote what sima said:
"another trick that we iirc used for afbc is that sometimes the planes 
have a fixed layout
like nv12
and so logically it's multiple planes, but you only need one plane slot 
to describe the buffer
since I think afbc had the "we need more than 4 planes" issue too"

Unfortunately, there are vendor pixel formats are not fixed layout.

3. Secure(REE, trusted video piepline) info.

For how to assign such metadata data.
In case with a drm fb_id, it is simple, we just add a drm plane property 
for it. The V4L2 interface is not flexible, we could only leave into 
CAPTURE request_fd as a control.
>>
>> Also, there is no reason to consume a device's memory for the content
>> that device can't read it, or wasting an entry of IOMMU for such data.
> 
> That's right, but DMA-buf doesn't really imply any of those. DMA-buf
> is just a kernel object with some backing memory. It's up to the
> allocator to decide how the backing memory is allocated and up to the
> importer on whether it would be mapped into an IOMMU.
> 
I just want to say it can't be allocated at the same place which was for 
those DMA bufs(graphics or compressed bitstream).
This also could be answer for your first question, if we place this kind 
of buffer in a plane for DMABUF(importing) in V4L2, V4L2 core would try 
to prepare it, which could map it into IOMMU.

>> Usually, such a metadata would be the value should be written to a
>> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
>>
>> Still, I have some problems with SHMEM:
>> 1. I don't want the userspace modify the context of the SHMEM allocated
>> by the kernel, is there a way to do so?
> 
> This is generally impossible without doing any of the two:
> 1) copying the contents to an internal buffer not accessible to the
> userspace, OR
> 2) modifying any of the buffer mappings to read-only
> 
> 2) can actually be more costly than 1) (depending on the architecture,
> data size, etc.), so we shouldn't just discard the option of a simple
> copy_from_user() in the ioctl.
> 
I don't want the userspace access it at all. So that won't be a problem.
>> 2. Should I create a helper function for installing the SHMEM file as a fd?
> 
> We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
> so maybe that would be enough?
> 
> [1] https://elixir.bootlin.com/linux/v6.5-rc7/source/drivers/dma-buf/udmabuf.c
> 
It is the kernel driver that allocate this buffer. For example, v4l2 
CAPTURE allocate a buffer for metadata when VIDIOC_REQBUFS.
Or GBM give you a fd which is assigned with a surface.

So we need a kernel interface.
> Best,
> Tomasz
> 
>>
>> --
>> Hsia-Jun(Randy) Li

-- 
Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Tomasz Figa]

On Wed, Aug 23, 2023 at 4:11 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
>
>
>
> On 8/23/23 12:46, Tomasz Figa wrote:
> > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> >
> > Hi Hsia-Jun,
> >
> > On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
> >>
> >> Hello
> >>
> >> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
> >> purpose of that is sharing metadata or just a pure container for cross
> >> drivers.
> >>
> >> We need to exchange some sort of metadata between drivers, likes dynamic
> >> HDR data between video4linux2 and DRM.
> >
> > If the metadata isn't too big, would it be enough to just have the
> > kernel copy_from_user() to a kernel buffer in the ioctl code?
> >
> >> Or the graphics frame buffer is
> >> too complex to be described with plain plane's DMA-buf fd.
> >> An issue between DRM and V4L2 is that DRM could only support 4 planes
> >> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
> >> interface to support that 4 more planes which would lead to revision of
> >> many standard likes Vulkan, EGL.
> >
> > Could you explain how a shmem buffer could be used to support frame
> > buffers with more than 4 planes?
> > If you are asking why we need this:

I'm asking how your proposal to use shmem FD solves the problem for those cases.

> 1. metadata likes dynamic HDR tone data
> 2. DRM also challenges with this problem, let me quote what sima said:
> "another trick that we iirc used for afbc is that sometimes the planes
> have a fixed layout
> like nv12
> and so logically it's multiple planes, but you only need one plane slot
> to describe the buffer
> since I think afbc had the "we need more than 4 planes" issue too"
>
> Unfortunately, there are vendor pixel formats are not fixed layout.
>
> 3. Secure(REE, trusted video piepline) info.
>
> For how to assign such metadata data.
> In case with a drm fb_id, it is simple, we just add a drm plane property
> for it. The V4L2 interface is not flexible, we could only leave into
> CAPTURE request_fd as a control.
> >>
> >> Also, there is no reason to consume a device's memory for the content
> >> that device can't read it, or wasting an entry of IOMMU for such data.
> >
> > That's right, but DMA-buf doesn't really imply any of those. DMA-buf
> > is just a kernel object with some backing memory. It's up to the
> > allocator to decide how the backing memory is allocated and up to the
> > importer on whether it would be mapped into an IOMMU.
> >
> I just want to say it can't be allocated at the same place which was for
> those DMA bufs(graphics or compressed bitstream).
> This also could be answer for your first question, if we place this kind
> of buffer in a plane for DMABUF(importing) in V4L2, V4L2 core would try
> to prepare it, which could map it into IOMMU.
>

V4L2 core will prepare it according to the struct device that is given
to it. For the planes that don't have to go to the hardware a struct
device could be given that doesn't require any DMA mapping. Also you
can check how the uvcvideo driver handles it. It doesn't use the vb2
buffers directly, but always writes to them using CPU (due to how the
UVC protocol is designed).

> >> Usually, such a metadata would be the value should be written to a
> >> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
> >>
> >> Still, I have some problems with SHMEM:
> >> 1. I don't want the userspace modify the context of the SHMEM allocated
> >> by the kernel, is there a way to do so?
> >
> > This is generally impossible without doing any of the two:
> > 1) copying the contents to an internal buffer not accessible to the
> > userspace, OR
> > 2) modifying any of the buffer mappings to read-only
> >
> > 2) can actually be more costly than 1) (depending on the architecture,
> > data size, etc.), so we shouldn't just discard the option of a simple
> > copy_from_user() in the ioctl.
> >
> I don't want the userspace access it at all. So that won't be a problem.

In this case, wouldn't it be enough to have a DMA-buf exporter that
doesn't provide the mmap op?

> >> 2. Should I create a helper function for installing the SHMEM file as a fd?
> >
> > We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
> > so maybe that would be enough?
> >
> > [1] https://elixir.bootlin.com/linux/v6.5-rc7/source/drivers/dma-buf/udmabuf.c
> >
> It is the kernel driver that allocate this buffer. For example, v4l2
> CAPTURE allocate a buffer for metadata when VIDIOC_REQBUFS.
> Or GBM give you a fd which is assigned with a surface.
>
> So we need a kernel interface.

Sorry, I'm confused. If we're talking about buffers allocated by the
specific allocators like V4L2 or GBM, why do we need SHMEM at all?

Best,
Tomasz

> > Best,
> > Tomasz
> >
> >>
> >> --
> >> Hsia-Jun(Randy) Li
>
> --
> Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Tomasz Figa]

On Wed, Aug 23, 2023 at 4:11 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
>
>
>
> On 8/23/23 12:46, Tomasz Figa wrote:
> > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> >
> > Hi Hsia-Jun,
> >
> > On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
> >>
> >> Hello
> >>
> >> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
> >> purpose of that is sharing metadata or just a pure container for cross
> >> drivers.
> >>
> >> We need to exchange some sort of metadata between drivers, likes dynamic
> >> HDR data between video4linux2 and DRM.
> >
> > If the metadata isn't too big, would it be enough to just have the
> > kernel copy_from_user() to a kernel buffer in the ioctl code?
> >
> >> Or the graphics frame buffer is
> >> too complex to be described with plain plane's DMA-buf fd.
> >> An issue between DRM and V4L2 is that DRM could only support 4 planes
> >> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
> >> interface to support that 4 more planes which would lead to revision of
> >> many standard likes Vulkan, EGL.
> >
> > Could you explain how a shmem buffer could be used to support frame
> > buffers with more than 4 planes?
> > If you are asking why we need this:

I'm asking how your proposal to use shmem FD solves the problem for those cases.

> 1. metadata likes dynamic HDR tone data
> 2. DRM also challenges with this problem, let me quote what sima said:
> "another trick that we iirc used for afbc is that sometimes the planes
> have a fixed layout
> like nv12
> and so logically it's multiple planes, but you only need one plane slot
> to describe the buffer
> since I think afbc had the "we need more than 4 planes" issue too"
>
> Unfortunately, there are vendor pixel formats are not fixed layout.
>
> 3. Secure(REE, trusted video piepline) info.
>
> For how to assign such metadata data.
> In case with a drm fb_id, it is simple, we just add a drm plane property
> for it. The V4L2 interface is not flexible, we could only leave into
> CAPTURE request_fd as a control.
> >>
> >> Also, there is no reason to consume a device's memory for the content
> >> that device can't read it, or wasting an entry of IOMMU for such data.
> >
> > That's right, but DMA-buf doesn't really imply any of those. DMA-buf
> > is just a kernel object with some backing memory. It's up to the
> > allocator to decide how the backing memory is allocated and up to the
> > importer on whether it would be mapped into an IOMMU.
> >
> I just want to say it can't be allocated at the same place which was for
> those DMA bufs(graphics or compressed bitstream).
> This also could be answer for your first question, if we place this kind
> of buffer in a plane for DMABUF(importing) in V4L2, V4L2 core would try
> to prepare it, which could map it into IOMMU.
>

V4L2 core will prepare it according to the struct device that is given
to it. For the planes that don't have to go to the hardware a struct
device could be given that doesn't require any DMA mapping. Also you
can check how the uvcvideo driver handles it. It doesn't use the vb2
buffers directly, but always writes to them using CPU (due to how the
UVC protocol is designed).

> >> Usually, such a metadata would be the value should be written to a
> >> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
> >>
> >> Still, I have some problems with SHMEM:
> >> 1. I don't want the userspace modify the context of the SHMEM allocated
> >> by the kernel, is there a way to do so?
> >
> > This is generally impossible without doing any of the two:
> > 1) copying the contents to an internal buffer not accessible to the
> > userspace, OR
> > 2) modifying any of the buffer mappings to read-only
> >
> > 2) can actually be more costly than 1) (depending on the architecture,
> > data size, etc.), so we shouldn't just discard the option of a simple
> > copy_from_user() in the ioctl.
> >
> I don't want the userspace access it at all. So that won't be a problem.

In this case, wouldn't it be enough to have a DMA-buf exporter that
doesn't provide the mmap op?

> >> 2. Should I create a helper function for installing the SHMEM file as a fd?
> >
> > We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
> > so maybe that would be enough?
> >
> > [1] https://elixir.bootlin.com/linux/v6.5-rc7/source/drivers/dma-buf/udmabuf.c
> >
> It is the kernel driver that allocate this buffer. For example, v4l2
> CAPTURE allocate a buffer for metadata when VIDIOC_REQBUFS.
> Or GBM give you a fd which is assigned with a surface.
>
> So we need a kernel interface.

Sorry, I'm confused. If we're talking about buffers allocated by the
specific allocators like V4L2 or GBM, why do we need SHMEM at all?

Best,
Tomasz

> > Best,
> > Tomasz
> >
> >>
> >> --
> >> Hsia-Jun(Randy) Li
>
> --
> Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Hsia-Jun Li]

On 8/23/23 21:15, Tomasz Figa wrote:
> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> On Wed, Aug 23, 2023 at 4:11 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
>>
>>
>>
>> On 8/23/23 12:46, Tomasz Figa wrote:
>>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>>
>>> Hi Hsia-Jun,
>>>
>>> On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
>>>>
>>>> Hello
>>>>
>>>> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
>>>> purpose of that is sharing metadata or just a pure container for cross
>>>> drivers.
>>>>
>>>> We need to exchange some sort of metadata between drivers, likes dynamic
>>>> HDR data between video4linux2 and DRM.
>>>
>>> If the metadata isn't too big, would it be enough to just have the
>>> kernel copy_from_user() to a kernel buffer in the ioctl code?
>>>
>>>> Or the graphics frame buffer is
>>>> too complex to be described with plain plane's DMA-buf fd.
>>>> An issue between DRM and V4L2 is that DRM could only support 4 planes
>>>> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
>>>> interface to support that 4 more planes which would lead to revision of
>>>> many standard likes Vulkan, EGL.
>>>
>>> Could you explain how a shmem buffer could be used to support frame
>>> buffers with more than 4 planes?
>>> If you are asking why we need this:
> 
> I'm asking how your proposal to use shmem FD solves the problem for those cases.
> 
The shmem fd is the reference to a metadata container(A C struct in 
kernel). Then drivers(V4L2 and DRM) could read this metadata when it 
processes the major buffer(SHMEM buf is the buffer assigned with a major 
buffer like the graphics buffer).
>> 1. metadata likes dynamic HDR tone data
>> 2. DRM also challenges with this problem, let me quote what sima said:
>> "another trick that we iirc used for afbc is that sometimes the planes
>> have a fixed layout
>> like nv12
>> and so logically it's multiple planes, but you only need one plane slot
>> to describe the buffer
>> since I think afbc had the "we need more than 4 planes" issue too"
>>
>> Unfortunately, there are vendor pixel formats are not fixed layout.
>>
>> 3. Secure(REE, trusted video piepline) info.
>>
>> For how to assign such metadata data.
>> In case with a drm fb_id, it is simple, we just add a drm plane property
>> for it. The V4L2 interface is not flexible, we could only leave into
>> CAPTURE request_fd as a control.
>>>>
>>>> Also, there is no reason to consume a device's memory for the content
>>>> that device can't read it, or wasting an entry of IOMMU for such data.
>>>
>>> That's right, but DMA-buf doesn't really imply any of those. DMA-buf
>>> is just a kernel object with some backing memory. It's up to the
>>> allocator to decide how the backing memory is allocated and up to the
>>> importer on whether it would be mapped into an IOMMU.
>>>
>> I just want to say it can't be allocated at the same place which was for
>> those DMA bufs(graphics or compressed bitstream).
>> This also could be answer for your first question, if we place this kind
>> of buffer in a plane for DMABUF(importing) in V4L2, V4L2 core would try
>> to prepare it, which could map it into IOMMU.
>>
> 
> V4L2 core will prepare it according to the struct device that is given
> to it. For the planes that don't have to go to the hardware a struct
> device could be given that doesn't require any DMA mapping. Also you
> can check how the uvcvideo driver handles it. It doesn't use the vb2
Because it uses vb2_vmalloc_memops?
That vb2_vmalloc_attach_dmabuf() won't work anything.
> buffers directly, but always writes to them using CPU (due to how the
Yes I noticed it would copy UBR buffer to vb2 buffer.
> UVC protocol is designed).
I don't know what stops that, because we can't assume xHCI or EHCI have 
the IOMMU?
> 
I think that is not I want. If you were not talking about META_CAPTURE, 
which would be a ioslated dev node.
For example, we have a NV15(2 planes) buffer with its HDR data.
We need its NV15 planes be accessed by DMA directly or it would be a 
performance issue(so UVC memcpy is not acceptable), while its HDR data, 
we just read it from the devices' registers or somewhere, should be ship 
with the exactly buffer.

Even we could expand the vb2_mem_ops interfaces, making it know which 
plane(ex. plane 0, 1 are graphics plane 2 is the metadata). The purpose 
here it not invoke the metadata buffer with any DMA buffer procedure.
>>>> Usually, such a metadata would be the value should be written to a
>>>> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
>>>>
>>>> Still, I have some problems with SHMEM:
>>>> 1. I don't want the userspace modify the context of the SHMEM allocated
>>>> by the kernel, is there a way to do so?
>>>
>>> This is generally impossible without doing any of the two:
>>> 1) copying the contents to an internal buffer not accessible to the
>>> userspace, OR
>>> 2) modifying any of the buffer mappings to read-only
>>>
>>> 2) can actually be more costly than 1) (depending on the architecture,
>>> data size, etc.), so we shouldn't just discard the option of a simple
>>> copy_from_user() in the ioctl.
>>>
>> I don't want the userspace access it at all. So that won't be a problem.
> 
> In this case, wouldn't it be enough to have a DMA-buf exporter that
> doesn't provide the mmap op?
> 
Also we want it allocate from vb2_mem_ops->alloc(), we could set an 
alloc_dev for a plane in queue_setup(), but where the metadata plane is 
would depends on the pixel formats.
It would be better not in vb2_planes.
>>>> 2. Should I create a helper function for installing the SHMEM file as a fd?
>>>
>>> We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
>>> so maybe that would be enough?
>>>
>>> [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__elixir.bootlin.com_linux_v6.5-2Drc7_source_drivers_dma-2Dbuf_udmabuf.c&d=DwIFaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=P4xb2_7biqBxD4LGGPrSV6j-jf3C3xlR7PXU-mLTeZE&m=Axkxip3NAR4d0Qb1sF508Emp922GJVs1gxv_UOP_DSxMMblD5iMl_F5-MiGyjiuZ&s=moL165MraNoXAqwiOfsw2qTeqLEzJ17Ut3J8DBx783c&e=
>>>
>> It is the kernel driver that allocate this buffer. For example, v4l2
>> CAPTURE allocate a buffer for metadata when VIDIOC_REQBUFS.
>> Or GBM give you a fd which is assigned with a surface.
>>
>> So we need a kernel interface.
> 
> Sorry, I'm confused. If we're talking about buffers allocated by the
> specific allocators like V4L2 or GBM, why do we need SHMEM at all?
> 
I would be in the IRC channel 5 hours later and this weekend, if 
anything is confusing, we could talk there.
> Best,
> Tomasz
> 
>>> Best,
>>> Tomasz
>>>
>>>>
>>>> --
>>>> Hsia-Jun(Randy) Li
>>
>> --
>> Hsia-Jun(Randy) Li

-- 
Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Hsia-Jun Li]

On 8/23/23 21:15, Tomasz Figa wrote:
> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> On Wed, Aug 23, 2023 at 4:11 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
>>
>>
>>
>> On 8/23/23 12:46, Tomasz Figa wrote:
>>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>>
>>> Hi Hsia-Jun,
>>>
>>> On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
>>>>
>>>> Hello
>>>>
>>>> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
>>>> purpose of that is sharing metadata or just a pure container for cross
>>>> drivers.
>>>>
>>>> We need to exchange some sort of metadata between drivers, likes dynamic
>>>> HDR data between video4linux2 and DRM.
>>>
>>> If the metadata isn't too big, would it be enough to just have the
>>> kernel copy_from_user() to a kernel buffer in the ioctl code?
>>>
>>>> Or the graphics frame buffer is
>>>> too complex to be described with plain plane's DMA-buf fd.
>>>> An issue between DRM and V4L2 is that DRM could only support 4 planes
>>>> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
>>>> interface to support that 4 more planes which would lead to revision of
>>>> many standard likes Vulkan, EGL.
>>>
>>> Could you explain how a shmem buffer could be used to support frame
>>> buffers with more than 4 planes?
>>> If you are asking why we need this:
> 
> I'm asking how your proposal to use shmem FD solves the problem for those cases.
> 
The shmem fd is the reference to a metadata container(A C struct in 
kernel). Then drivers(V4L2 and DRM) could read this metadata when it 
processes the major buffer(SHMEM buf is the buffer assigned with a major 
buffer like the graphics buffer).
>> 1. metadata likes dynamic HDR tone data
>> 2. DRM also challenges with this problem, let me quote what sima said:
>> "another trick that we iirc used for afbc is that sometimes the planes
>> have a fixed layout
>> like nv12
>> and so logically it's multiple planes, but you only need one plane slot
>> to describe the buffer
>> since I think afbc had the "we need more than 4 planes" issue too"
>>
>> Unfortunately, there are vendor pixel formats are not fixed layout.
>>
>> 3. Secure(REE, trusted video piepline) info.
>>
>> For how to assign such metadata data.
>> In case with a drm fb_id, it is simple, we just add a drm plane property
>> for it. The V4L2 interface is not flexible, we could only leave into
>> CAPTURE request_fd as a control.
>>>>
>>>> Also, there is no reason to consume a device's memory for the content
>>>> that device can't read it, or wasting an entry of IOMMU for such data.
>>>
>>> That's right, but DMA-buf doesn't really imply any of those. DMA-buf
>>> is just a kernel object with some backing memory. It's up to the
>>> allocator to decide how the backing memory is allocated and up to the
>>> importer on whether it would be mapped into an IOMMU.
>>>
>> I just want to say it can't be allocated at the same place which was for
>> those DMA bufs(graphics or compressed bitstream).
>> This also could be answer for your first question, if we place this kind
>> of buffer in a plane for DMABUF(importing) in V4L2, V4L2 core would try
>> to prepare it, which could map it into IOMMU.
>>
> 
> V4L2 core will prepare it according to the struct device that is given
> to it. For the planes that don't have to go to the hardware a struct
> device could be given that doesn't require any DMA mapping. Also you
> can check how the uvcvideo driver handles it. It doesn't use the vb2
Because it uses vb2_vmalloc_memops?
That vb2_vmalloc_attach_dmabuf() won't work anything.
> buffers directly, but always writes to them using CPU (due to how the
Yes I noticed it would copy UBR buffer to vb2 buffer.
> UVC protocol is designed).
I don't know what stops that, because we can't assume xHCI or EHCI have 
the IOMMU?
> 
I think that is not I want. If you were not talking about META_CAPTURE, 
which would be a ioslated dev node.
For example, we have a NV15(2 planes) buffer with its HDR data.
We need its NV15 planes be accessed by DMA directly or it would be a 
performance issue(so UVC memcpy is not acceptable), while its HDR data, 
we just read it from the devices' registers or somewhere, should be ship 
with the exactly buffer.

Even we could expand the vb2_mem_ops interfaces, making it know which 
plane(ex. plane 0, 1 are graphics plane 2 is the metadata). The purpose 
here it not invoke the metadata buffer with any DMA buffer procedure.
>>>> Usually, such a metadata would be the value should be written to a
>>>> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
>>>>
>>>> Still, I have some problems with SHMEM:
>>>> 1. I don't want the userspace modify the context of the SHMEM allocated
>>>> by the kernel, is there a way to do so?
>>>
>>> This is generally impossible without doing any of the two:
>>> 1) copying the contents to an internal buffer not accessible to the
>>> userspace, OR
>>> 2) modifying any of the buffer mappings to read-only
>>>
>>> 2) can actually be more costly than 1) (depending on the architecture,
>>> data size, etc.), so we shouldn't just discard the option of a simple
>>> copy_from_user() in the ioctl.
>>>
>> I don't want the userspace access it at all. So that won't be a problem.
> 
> In this case, wouldn't it be enough to have a DMA-buf exporter that
> doesn't provide the mmap op?
> 
Also we want it allocate from vb2_mem_ops->alloc(), we could set an 
alloc_dev for a plane in queue_setup(), but where the metadata plane is 
would depends on the pixel formats.
It would be better not in vb2_planes.
>>>> 2. Should I create a helper function for installing the SHMEM file as a fd?
>>>
>>> We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
>>> so maybe that would be enough?
>>>
>>> [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__elixir.bootlin.com_linux_v6.5-2Drc7_source_drivers_dma-2Dbuf_udmabuf.c&d=DwIFaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=P4xb2_7biqBxD4LGGPrSV6j-jf3C3xlR7PXU-mLTeZE&m=Axkxip3NAR4d0Qb1sF508Emp922GJVs1gxv_UOP_DSxMMblD5iMl_F5-MiGyjiuZ&s=moL165MraNoXAqwiOfsw2qTeqLEzJ17Ut3J8DBx783c&e=
>>>
>> It is the kernel driver that allocate this buffer. For example, v4l2
>> CAPTURE allocate a buffer for metadata when VIDIOC_REQBUFS.
>> Or GBM give you a fd which is assigned with a surface.
>>
>> So we need a kernel interface.
> 
> Sorry, I'm confused. If we're talking about buffers allocated by the
> specific allocators like V4L2 or GBM, why do we need SHMEM at all?
> 
I would be in the IRC channel 5 hours later and this weekend, if 
anything is confusing, we could talk there.
> Best,
> Tomasz
> 
>>> Best,
>>> Tomasz
>>>
>>>>
>>>> --
>>>> Hsia-Jun(Randy) Li
>>
>> --
>> Hsia-Jun(Randy) Li

-- 
Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 5742 bytes --]

On Wed, 23 Aug 2023 15:11:23 +0800
Hsia-Jun Li <Randy.Li@synaptics.com> wrote:

> On 8/23/23 12:46, Tomasz Figa wrote:
> > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> > 
> > 
> > Hi Hsia-Jun,
> > 
> > On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:  
> >>
> >> Hello
> >>
> >> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
> >> purpose of that is sharing metadata or just a pure container for cross
> >> drivers.
> >>
> >> We need to exchange some sort of metadata between drivers, likes dynamic
> >> HDR data between video4linux2 and DRM.  
> > 
> > If the metadata isn't too big, would it be enough to just have the
> > kernel copy_from_user() to a kernel buffer in the ioctl code?
> >   
> >> Or the graphics frame buffer is
> >> too complex to be described with plain plane's DMA-buf fd.
> >> An issue between DRM and V4L2 is that DRM could only support 4 planes
> >> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
> >> interface to support that 4 more planes which would lead to revision of
> >> many standard likes Vulkan, EGL.  
> > 
> > Could you explain how a shmem buffer could be used to support frame
> > buffers with more than 4 planes?
> > If you are asking why we need this:  
> 1. metadata likes dynamic HDR tone data
> 2. DRM also challenges with this problem, let me quote what sima said:
> "another trick that we iirc used for afbc is that sometimes the planes 
> have a fixed layout
> like nv12
> and so logically it's multiple planes, but you only need one plane slot 
> to describe the buffer
> since I think afbc had the "we need more than 4 planes" issue too"
> 
> Unfortunately, there are vendor pixel formats are not fixed layout.
> 
> 3. Secure(REE, trusted video piepline) info.
> 
> For how to assign such metadata data.
> In case with a drm fb_id, it is simple, we just add a drm plane property 
> for it. The V4L2 interface is not flexible, we could only leave into 
> CAPTURE request_fd as a control.
> >>
> >> Also, there is no reason to consume a device's memory for the content
> >> that device can't read it, or wasting an entry of IOMMU for such data.  
> > 
> > That's right, but DMA-buf doesn't really imply any of those. DMA-buf
> > is just a kernel object with some backing memory. It's up to the
> > allocator to decide how the backing memory is allocated and up to the
> > importer on whether it would be mapped into an IOMMU.
> >   
> I just want to say it can't be allocated at the same place which was for 
> those DMA bufs(graphics or compressed bitstream).
> This also could be answer for your first question, if we place this kind 
> of buffer in a plane for DMABUF(importing) in V4L2, V4L2 core would try 
> to prepare it, which could map it into IOMMU.
> 
> >> Usually, such a metadata would be the value should be written to a
> >> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
> >>
> >> Still, I have some problems with SHMEM:
> >> 1. I don't want the userspace modify the context of the SHMEM allocated
> >> by the kernel, is there a way to do so?  
> > 
> > This is generally impossible without doing any of the two:
> > 1) copying the contents to an internal buffer not accessible to the
> > userspace, OR
> > 2) modifying any of the buffer mappings to read-only
> > 
> > 2) can actually be more costly than 1) (depending on the architecture,
> > data size, etc.), so we shouldn't just discard the option of a simple
> > copy_from_user() in the ioctl.
> >   
> I don't want the userspace access it at all. So that won't be a problem.

Hi,

if userspace cannot access things like an image's HDR metadata, then it
will be impossible for userspace to program KMS to have the correct
color pipeline, or to send intended HDR metadata to a video sink.

You cannot leave userspace out of HDR metadata handling, because quite
probably the V4L2 buffer is not the only thing on screen. That means
there must composition of multiple sources with different image
properties and metadata, which means it is no longer obvious what HDR
metadata should be sent to the video sink.

Even if it is a TV-like application rather than a windowed desktop, you
will still have other contents to composite: OSD (volume indicators,
channels indicators, program guide, ...), sub-titles, channel logos,
notifications... These components ideally should not change their
appearance arbitrarily with the main program content and metadata
changes. Either the metadata sent to the video sink is kept static and
the main program adapted on the fly, or main program metadata is sent
to the video sink and the additional content is adapted on the fly.

There is only one set of HDR metadata and one composited image that can
be sent to a video sink, so both must be chosen and produced correctly
at the source side. This cannot be done automatically inside KMS kernel
drivers.


Thanks,
pq

> >> 2. Should I create a helper function for installing the SHMEM file as a fd?  
> > 
> > We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
> > so maybe that would be enough?
> > 
> > [1] https://elixir.bootlin.com/linux/v6.5-rc7/source/drivers/dma-buf/udmabuf.c
> >   
> It is the kernel driver that allocate this buffer. For example, v4l2 
> CAPTURE allocate a buffer for metadata when VIDIOC_REQBUFS.
> Or GBM give you a fd which is assigned with a surface.
> 
> So we need a kernel interface.
> > Best,
> > Tomasz
> >   
> >>
> >> --
> >> Hsia-Jun(Randy) Li  
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 5742 bytes --]

On Wed, 23 Aug 2023 15:11:23 +0800
Hsia-Jun Li <Randy.Li@synaptics.com> wrote:

> On 8/23/23 12:46, Tomasz Figa wrote:
> > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> > 
> > 
> > Hi Hsia-Jun,
> > 
> > On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote:  
> >>
> >> Hello
> >>
> >> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
> >> purpose of that is sharing metadata or just a pure container for cross
> >> drivers.
> >>
> >> We need to exchange some sort of metadata between drivers, likes dynamic
> >> HDR data between video4linux2 and DRM.  
> > 
> > If the metadata isn't too big, would it be enough to just have the
> > kernel copy_from_user() to a kernel buffer in the ioctl code?
> >   
> >> Or the graphics frame buffer is
> >> too complex to be described with plain plane's DMA-buf fd.
> >> An issue between DRM and V4L2 is that DRM could only support 4 planes
> >> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
> >> interface to support that 4 more planes which would lead to revision of
> >> many standard likes Vulkan, EGL.  
> > 
> > Could you explain how a shmem buffer could be used to support frame
> > buffers with more than 4 planes?
> > If you are asking why we need this:  
> 1. metadata likes dynamic HDR tone data
> 2. DRM also challenges with this problem, let me quote what sima said:
> "another trick that we iirc used for afbc is that sometimes the planes 
> have a fixed layout
> like nv12
> and so logically it's multiple planes, but you only need one plane slot 
> to describe the buffer
> since I think afbc had the "we need more than 4 planes" issue too"
> 
> Unfortunately, there are vendor pixel formats are not fixed layout.
> 
> 3. Secure(REE, trusted video piepline) info.
> 
> For how to assign such metadata data.
> In case with a drm fb_id, it is simple, we just add a drm plane property 
> for it. The V4L2 interface is not flexible, we could only leave into 
> CAPTURE request_fd as a control.
> >>
> >> Also, there is no reason to consume a device's memory for the content
> >> that device can't read it, or wasting an entry of IOMMU for such data.  
> > 
> > That's right, but DMA-buf doesn't really imply any of those. DMA-buf
> > is just a kernel object with some backing memory. It's up to the
> > allocator to decide how the backing memory is allocated and up to the
> > importer on whether it would be mapped into an IOMMU.
> >   
> I just want to say it can't be allocated at the same place which was for 
> those DMA bufs(graphics or compressed bitstream).
> This also could be answer for your first question, if we place this kind 
> of buffer in a plane for DMABUF(importing) in V4L2, V4L2 core would try 
> to prepare it, which could map it into IOMMU.
> 
> >> Usually, such a metadata would be the value should be written to a
> >> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
> >>
> >> Still, I have some problems with SHMEM:
> >> 1. I don't want the userspace modify the context of the SHMEM allocated
> >> by the kernel, is there a way to do so?  
> > 
> > This is generally impossible without doing any of the two:
> > 1) copying the contents to an internal buffer not accessible to the
> > userspace, OR
> > 2) modifying any of the buffer mappings to read-only
> > 
> > 2) can actually be more costly than 1) (depending on the architecture,
> > data size, etc.), so we shouldn't just discard the option of a simple
> > copy_from_user() in the ioctl.
> >   
> I don't want the userspace access it at all. So that won't be a problem.

Hi,

if userspace cannot access things like an image's HDR metadata, then it
will be impossible for userspace to program KMS to have the correct
color pipeline, or to send intended HDR metadata to a video sink.

You cannot leave userspace out of HDR metadata handling, because quite
probably the V4L2 buffer is not the only thing on screen. That means
there must composition of multiple sources with different image
properties and metadata, which means it is no longer obvious what HDR
metadata should be sent to the video sink.

Even if it is a TV-like application rather than a windowed desktop, you
will still have other contents to composite: OSD (volume indicators,
channels indicators, program guide, ...), sub-titles, channel logos,
notifications... These components ideally should not change their
appearance arbitrarily with the main program content and metadata
changes. Either the metadata sent to the video sink is kept static and
the main program adapted on the fly, or main program metadata is sent
to the video sink and the additional content is adapted on the fly.

There is only one set of HDR metadata and one composited image that can
be sent to a video sink, so both must be chosen and produced correctly
at the source side. This cannot be done automatically inside KMS kernel
drivers.


Thanks,
pq

> >> 2. Should I create a helper function for installing the SHMEM file as a fd?  
> > 
> > We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
> > so maybe that would be enough?
> > 
> > [1] https://elixir.bootlin.com/linux/v6.5-rc7/source/drivers/dma-buf/udmabuf.c
> >   
> It is the kernel driver that allocate this buffer. For example, v4l2 
> CAPTURE allocate a buffer for metadata when VIDIOC_REQBUFS.
> Or GBM give you a fd which is assigned with a surface.
> 
> So we need a kernel interface.
> > Best,
> > Tomasz
> >   
> >>
> >> --
> >> Hsia-Jun(Randy) Li  
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Hsia-Jun Li]

On 8/25/23 15:40, Pekka Paalanen wrote:
> Subject:
> Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers
> From:
> Pekka Paalanen <ppaalanen@gmail.com>
> Date:
> 8/25/23, 15:40
> 
> To:
> Hsia-Jun Li <Randy.Li@synaptics.com>
> CC:
> Tomasz Figa <tfiga@chromium.org>, linux-mm@kvack.org, 
> dri-devel@lists.freedesktop.org, Linux Media Mailing List 
> <linux-media@vger.kernel.org>, hughd@google.com, 
> akpm@linux-foundation.org, Simon Ser <contact@emersion.fr>, Hans Verkuil 
> <hverkuil-cisco@xs4all.nl>, daniels@collabora.com, ayaka 
> <ayaka@soulik.info>, linux-kernel@vger.kernel.org, Nicolas Dufresne 
> <nicolas@ndufresne.ca>
> 
> 
> On Wed, 23 Aug 2023 15:11:23 +0800
> Hsia-Jun Li<Randy.Li@synaptics.com>  wrote:
> 
>> On 8/23/23 12:46, Tomasz Figa wrote:
>>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>>
>>> Hi Hsia-Jun,
>>>
>>> On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li<Randy.Li@synaptics.com>  wrote:
>>>> Hello
>>>>
>>>> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
>>>> purpose of that is sharing metadata or just a pure container for cross
>>>> drivers.
>>>>
>>>> We need to exchange some sort of metadata between drivers, likes dynamic
>>>> HDR data between video4linux2 and DRM.
>>> If the metadata isn't too big, would it be enough to just have the
>>> kernel copy_from_user() to a kernel buffer in the ioctl code?
>>>    
>>>> Or the graphics frame buffer is
>>>> too complex to be described with plain plane's DMA-buf fd.
>>>> An issue between DRM and V4L2 is that DRM could only support 4 planes
>>>> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
>>>> interface to support that 4 more planes which would lead to revision of
>>>> many standard likes Vulkan, EGL.
>>> Could you explain how a shmem buffer could be used to support frame
>>> buffers with more than 4 planes?
>>> If you are asking why we need this:
>> 1. metadata likes dynamic HDR tone data
>> 2. DRM also challenges with this problem, let me quote what sima said:
>> "another trick that we iirc used for afbc is that sometimes the planes
>> have a fixed layout
>> like nv12
>> and so logically it's multiple planes, but you only need one plane slot
>> to describe the buffer
>> since I think afbc had the "we need more than 4 planes" issue too"
>>
>> Unfortunately, there are vendor pixel formats are not fixed layout.
>>
>> 3. Secure(REE, trusted video piepline) info.
>>
>> For how to assign such metadata data.
>> In case with a drm fb_id, it is simple, we just add a drm plane property
>> for it. The V4L2 interface is not flexible, we could only leave into
>> CAPTURE request_fd as a control.
>>>> Also, there is no reason to consume a device's memory for the content
>>>> that device can't read it, or wasting an entry of IOMMU for such data.
>>> That's right, but DMA-buf doesn't really imply any of those. DMA-buf
>>> is just a kernel object with some backing memory. It's up to the
>>> allocator to decide how the backing memory is allocated and up to the
>>> importer on whether it would be mapped into an IOMMU.
>>>    
>> I just want to say it can't be allocated at the same place which was for
>> those DMA bufs(graphics or compressed bitstream).
>> This also could be answer for your first question, if we place this kind
>> of buffer in a plane for DMABUF(importing) in V4L2, V4L2 core would try
>> to prepare it, which could map it into IOMMU.
>>
>>>> Usually, such a metadata would be the value should be written to a
>>>> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
>>>>
>>>> Still, I have some problems with SHMEM:
>>>> 1. I don't want the userspace modify the context of the SHMEM allocated
>>>> by the kernel, is there a way to do so?
>>> This is generally impossible without doing any of the two:
>>> 1) copying the contents to an internal buffer not accessible to the
>>> userspace, OR
>>> 2) modifying any of the buffer mappings to read-only
>>>
>>> 2) can actually be more costly than 1) (depending on the architecture,
>>> data size, etc.), so we shouldn't just discard the option of a simple
>>> copy_from_user() in the ioctl.
>>>    
>> I don't want the userspace access it at all. So that won't be a problem.
> Hi,
> 
> if userspace cannot access things like an image's HDR metadata, then it
> will be impossible for userspace to program KMS to have the correct
> color pipeline, or to send intended HDR metadata to a video sink.
> 
> You cannot leave userspace out of HDR metadata handling, because quite
> probably the V4L2 buffer is not the only thing on screen. That means
> there must composition of multiple sources with different image
> properties and metadata, which means it is no longer obvious what HDR
> metadata should be sent to the video sink.
> 
> Even if it is a TV-like application rather than a windowed desktop, you
> will still have other contents to composite: OSD (volume indicators,
> channels indicators, program guide, ...), sub-titles, channel logos,
> notifications... These components ideally should not change their
> appearance arbitrarily with the main program content and metadata
> changes. Either the metadata sent to the video sink is kept static and
> the main program adapted on the fly, or main program metadata is sent
> to the video sink and the additional content is adapted on the fly.
> 
> There is only one set of HDR metadata and one composited image that can
> be sent to a video sink, so both must be chosen and produced correctly
> at the source side. This cannot be done automatically inside KMS kernel
> drivers.
> 
There may be some misunderstanding.
Let suppose this HDR data is in a vendor specific format.
Both upstream(decoder) and downstream(DRM) hardware devices are coming 
from the same vendor.
Then we just need to delivery the reference to this metadata buffer from 
the upstream to downstream, both of drivers know how to handle it.

Despite the userspace, we just need to extend a wayland protocol that 
making wayland compositor know how to receive the reference to the 
metadata and set it to the DRM plane.

If you want a common HDR formats for all HDR variants(HDR10+, DV), I am 
not against it. But it won't make the userspace be able to fill the HDR 
metadata even the HDR data comes from the bitstream(likes SEI). We must 
consider the case of Secure Video Path(Digital Right), the bitstream is 
not accessible from (REE) userspace nor linux kernel, the downstream 
must take what the upstream feed.
> 
> Thanks,
> pq
> 
>>>> 2. Should I create a helper function for installing the SHMEM file as a fd?
>>> We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
>>> so maybe that would be enough?
>>>
>>> [1]https://elixir.bootlin.com/linux/v6.5-rc7/source/drivers/dma-buf/udmabuf.c
>>>    
>> It is the kernel driver that allocate this buffer. For example, v4l2
>> CAPTURE allocate a buffer for metadata when VIDIOC_REQBUFS.
>> Or GBM give you a fd which is assigned with a surface.
>>
>> So we need a kernel interface.
>>> Best,
>>> Tomasz
>>>    
>>>> --
>>>> Hsia-Jun(Randy) Li
> 

-- 
Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Hsia-Jun Li]

On 8/25/23 15:40, Pekka Paalanen wrote:
> Subject:
> Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers
> From:
> Pekka Paalanen <ppaalanen@gmail.com>
> Date:
> 8/25/23, 15:40
> 
> To:
> Hsia-Jun Li <Randy.Li@synaptics.com>
> CC:
> Tomasz Figa <tfiga@chromium.org>, linux-mm@kvack.org, 
> dri-devel@lists.freedesktop.org, Linux Media Mailing List 
> <linux-media@vger.kernel.org>, hughd@google.com, 
> akpm@linux-foundation.org, Simon Ser <contact@emersion.fr>, Hans Verkuil 
> <hverkuil-cisco@xs4all.nl>, daniels@collabora.com, ayaka 
> <ayaka@soulik.info>, linux-kernel@vger.kernel.org, Nicolas Dufresne 
> <nicolas@ndufresne.ca>
> 
> 
> On Wed, 23 Aug 2023 15:11:23 +0800
> Hsia-Jun Li<Randy.Li@synaptics.com>  wrote:
> 
>> On 8/23/23 12:46, Tomasz Figa wrote:
>>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>>
>>> Hi Hsia-Jun,
>>>
>>> On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li<Randy.Li@synaptics.com>  wrote:
>>>> Hello
>>>>
>>>> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
>>>> purpose of that is sharing metadata or just a pure container for cross
>>>> drivers.
>>>>
>>>> We need to exchange some sort of metadata between drivers, likes dynamic
>>>> HDR data between video4linux2 and DRM.
>>> If the metadata isn't too big, would it be enough to just have the
>>> kernel copy_from_user() to a kernel buffer in the ioctl code?
>>>    
>>>> Or the graphics frame buffer is
>>>> too complex to be described with plain plane's DMA-buf fd.
>>>> An issue between DRM and V4L2 is that DRM could only support 4 planes
>>>> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
>>>> interface to support that 4 more planes which would lead to revision of
>>>> many standard likes Vulkan, EGL.
>>> Could you explain how a shmem buffer could be used to support frame
>>> buffers with more than 4 planes?
>>> If you are asking why we need this:
>> 1. metadata likes dynamic HDR tone data
>> 2. DRM also challenges with this problem, let me quote what sima said:
>> "another trick that we iirc used for afbc is that sometimes the planes
>> have a fixed layout
>> like nv12
>> and so logically it's multiple planes, but you only need one plane slot
>> to describe the buffer
>> since I think afbc had the "we need more than 4 planes" issue too"
>>
>> Unfortunately, there are vendor pixel formats are not fixed layout.
>>
>> 3. Secure(REE, trusted video piepline) info.
>>
>> For how to assign such metadata data.
>> In case with a drm fb_id, it is simple, we just add a drm plane property
>> for it. The V4L2 interface is not flexible, we could only leave into
>> CAPTURE request_fd as a control.
>>>> Also, there is no reason to consume a device's memory for the content
>>>> that device can't read it, or wasting an entry of IOMMU for such data.
>>> That's right, but DMA-buf doesn't really imply any of those. DMA-buf
>>> is just a kernel object with some backing memory. It's up to the
>>> allocator to decide how the backing memory is allocated and up to the
>>> importer on whether it would be mapped into an IOMMU.
>>>    
>> I just want to say it can't be allocated at the same place which was for
>> those DMA bufs(graphics or compressed bitstream).
>> This also could be answer for your first question, if we place this kind
>> of buffer in a plane for DMABUF(importing) in V4L2, V4L2 core would try
>> to prepare it, which could map it into IOMMU.
>>
>>>> Usually, such a metadata would be the value should be written to a
>>>> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
>>>>
>>>> Still, I have some problems with SHMEM:
>>>> 1. I don't want the userspace modify the context of the SHMEM allocated
>>>> by the kernel, is there a way to do so?
>>> This is generally impossible without doing any of the two:
>>> 1) copying the contents to an internal buffer not accessible to the
>>> userspace, OR
>>> 2) modifying any of the buffer mappings to read-only
>>>
>>> 2) can actually be more costly than 1) (depending on the architecture,
>>> data size, etc.), so we shouldn't just discard the option of a simple
>>> copy_from_user() in the ioctl.
>>>    
>> I don't want the userspace access it at all. So that won't be a problem.
> Hi,
> 
> if userspace cannot access things like an image's HDR metadata, then it
> will be impossible for userspace to program KMS to have the correct
> color pipeline, or to send intended HDR metadata to a video sink.
> 
> You cannot leave userspace out of HDR metadata handling, because quite
> probably the V4L2 buffer is not the only thing on screen. That means
> there must composition of multiple sources with different image
> properties and metadata, which means it is no longer obvious what HDR
> metadata should be sent to the video sink.
> 
> Even if it is a TV-like application rather than a windowed desktop, you
> will still have other contents to composite: OSD (volume indicators,
> channels indicators, program guide, ...), sub-titles, channel logos,
> notifications... These components ideally should not change their
> appearance arbitrarily with the main program content and metadata
> changes. Either the metadata sent to the video sink is kept static and
> the main program adapted on the fly, or main program metadata is sent
> to the video sink and the additional content is adapted on the fly.
> 
> There is only one set of HDR metadata and one composited image that can
> be sent to a video sink, so both must be chosen and produced correctly
> at the source side. This cannot be done automatically inside KMS kernel
> drivers.
> 
There may be some misunderstanding.
Let suppose this HDR data is in a vendor specific format.
Both upstream(decoder) and downstream(DRM) hardware devices are coming 
from the same vendor.
Then we just need to delivery the reference to this metadata buffer from 
the upstream to downstream, both of drivers know how to handle it.

Despite the userspace, we just need to extend a wayland protocol that 
making wayland compositor know how to receive the reference to the 
metadata and set it to the DRM plane.

If you want a common HDR formats for all HDR variants(HDR10+, DV), I am 
not against it. But it won't make the userspace be able to fill the HDR 
metadata even the HDR data comes from the bitstream(likes SEI). We must 
consider the case of Secure Video Path(Digital Right), the bitstream is 
not accessible from (REE) userspace nor linux kernel, the downstream 
must take what the upstream feed.
> 
> Thanks,
> pq
> 
>>>> 2. Should I create a helper function for installing the SHMEM file as a fd?
>>> We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
>>> so maybe that would be enough?
>>>
>>> [1]https://elixir.bootlin.com/linux/v6.5-rc7/source/drivers/dma-buf/udmabuf.c
>>>    
>> It is the kernel driver that allocate this buffer. For example, v4l2
>> CAPTURE allocate a buffer for metadata when VIDIOC_REQBUFS.
>> Or GBM give you a fd which is assigned with a surface.
>>
>> So we need a kernel interface.
>>> Best,
>>> Tomasz
>>>    
>>>> --
>>>> Hsia-Jun(Randy) Li
> 

-- 
Hsia-Jun(Randy) Li

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 8693 bytes --]

On Fri, 25 Aug 2023 15:56:18 +0800
Hsia-Jun Li <Randy.Li@synaptics.com> wrote:

> On 8/25/23 15:40, Pekka Paalanen wrote:
> > Subject:
> > Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers
> > From:
> > Pekka Paalanen <ppaalanen@gmail.com>
> > Date:
> > 8/25/23, 15:40
> > 
> > To:
> > Hsia-Jun Li <Randy.Li@synaptics.com>
> > CC:
> > Tomasz Figa <tfiga@chromium.org>, linux-mm@kvack.org, 
> > dri-devel@lists.freedesktop.org, Linux Media Mailing List 
> > <linux-media@vger.kernel.org>, hughd@google.com, 
> > akpm@linux-foundation.org, Simon Ser <contact@emersion.fr>, Hans Verkuil 
> > <hverkuil-cisco@xs4all.nl>, daniels@collabora.com, ayaka 
> > <ayaka@soulik.info>, linux-kernel@vger.kernel.org, Nicolas Dufresne 
> > <nicolas@ndufresne.ca>
> > 
> > 
> > On Wed, 23 Aug 2023 15:11:23 +0800
> > Hsia-Jun Li<Randy.Li@synaptics.com>  wrote:
> >   
> >> On 8/23/23 12:46, Tomasz Figa wrote:  
> >>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> >>>
> >>>
> >>> Hi Hsia-Jun,
> >>>
> >>> On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li<Randy.Li@synaptics.com>  wrote:  
> >>>> Hello
> >>>>
> >>>> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
> >>>> purpose of that is sharing metadata or just a pure container for cross
> >>>> drivers.
> >>>>
> >>>> We need to exchange some sort of metadata between drivers, likes dynamic
> >>>> HDR data between video4linux2 and DRM.  
> >>> If the metadata isn't too big, would it be enough to just have the
> >>> kernel copy_from_user() to a kernel buffer in the ioctl code?
> >>>      
> >>>> Or the graphics frame buffer is
> >>>> too complex to be described with plain plane's DMA-buf fd.
> >>>> An issue between DRM and V4L2 is that DRM could only support 4 planes
> >>>> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
> >>>> interface to support that 4 more planes which would lead to revision of
> >>>> many standard likes Vulkan, EGL.  
> >>> Could you explain how a shmem buffer could be used to support frame
> >>> buffers with more than 4 planes?
> >>> If you are asking why we need this:  
> >> 1. metadata likes dynamic HDR tone data
> >> 2. DRM also challenges with this problem, let me quote what sima said:
> >> "another trick that we iirc used for afbc is that sometimes the planes
> >> have a fixed layout
> >> like nv12
> >> and so logically it's multiple planes, but you only need one plane slot
> >> to describe the buffer
> >> since I think afbc had the "we need more than 4 planes" issue too"
> >>
> >> Unfortunately, there are vendor pixel formats are not fixed layout.
> >>
> >> 3. Secure(REE, trusted video piepline) info.
> >>
> >> For how to assign such metadata data.
> >> In case with a drm fb_id, it is simple, we just add a drm plane property
> >> for it. The V4L2 interface is not flexible, we could only leave into
> >> CAPTURE request_fd as a control.  
> >>>> Also, there is no reason to consume a device's memory for the content
> >>>> that device can't read it, or wasting an entry of IOMMU for such data.  
> >>> That's right, but DMA-buf doesn't really imply any of those. DMA-buf
> >>> is just a kernel object with some backing memory. It's up to the
> >>> allocator to decide how the backing memory is allocated and up to the
> >>> importer on whether it would be mapped into an IOMMU.
> >>>      
> >> I just want to say it can't be allocated at the same place which was for
> >> those DMA bufs(graphics or compressed bitstream).
> >> This also could be answer for your first question, if we place this kind
> >> of buffer in a plane for DMABUF(importing) in V4L2, V4L2 core would try
> >> to prepare it, which could map it into IOMMU.
> >>  
> >>>> Usually, such a metadata would be the value should be written to a
> >>>> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
> >>>>
> >>>> Still, I have some problems with SHMEM:
> >>>> 1. I don't want the userspace modify the context of the SHMEM allocated
> >>>> by the kernel, is there a way to do so?  
> >>> This is generally impossible without doing any of the two:
> >>> 1) copying the contents to an internal buffer not accessible to the
> >>> userspace, OR
> >>> 2) modifying any of the buffer mappings to read-only
> >>>
> >>> 2) can actually be more costly than 1) (depending on the architecture,
> >>> data size, etc.), so we shouldn't just discard the option of a simple
> >>> copy_from_user() in the ioctl.
> >>>      
> >> I don't want the userspace access it at all. So that won't be a problem.  
> > Hi,
> > 
> > if userspace cannot access things like an image's HDR metadata, then it
> > will be impossible for userspace to program KMS to have the correct
> > color pipeline, or to send intended HDR metadata to a video sink.
> > 
> > You cannot leave userspace out of HDR metadata handling, because quite
> > probably the V4L2 buffer is not the only thing on screen. That means
> > there must composition of multiple sources with different image
> > properties and metadata, which means it is no longer obvious what HDR
> > metadata should be sent to the video sink.
> > 
> > Even if it is a TV-like application rather than a windowed desktop, you
> > will still have other contents to composite: OSD (volume indicators,
> > channels indicators, program guide, ...), sub-titles, channel logos,
> > notifications... These components ideally should not change their
> > appearance arbitrarily with the main program content and metadata
> > changes. Either the metadata sent to the video sink is kept static and
> > the main program adapted on the fly, or main program metadata is sent
> > to the video sink and the additional content is adapted on the fly.
> > 
> > There is only one set of HDR metadata and one composited image that can
> > be sent to a video sink, so both must be chosen and produced correctly
> > at the source side. This cannot be done automatically inside KMS kernel
> > drivers.
> >   
> There may be some misunderstanding.
> Let suppose this HDR data is in a vendor specific format.
> Both upstream(decoder) and downstream(DRM) hardware devices are coming 
> from the same vendor.
> Then we just need to delivery the reference to this metadata buffer from 
> the upstream to downstream, both of drivers know how to handle it.
> 
> Despite the userspace, we just need to extend a wayland protocol that 
> making wayland compositor know how to receive the reference to the 
> metadata and set it to the DRM plane.

Hi,

FWIW, I don't think Wayland upstream (wayland-protocols) would be any
more welcoming to opaque vendor-specific proprietary data blobs than the
upstream Linux.

(To everyone not following #dri-devel: There was a fairly long IRC
discussion about how HDR metadata is an exceptionally poor use case
example for "hidden" ancillary data blobs, and then continuing to how a
mechanism to pass proprietary data blobs around would not be accepted
upstream:
https://oftc.irclog.whitequark.org/dri-devel/2023-08-25#1692950883-1692960435;
)

> If you want a common HDR formats for all HDR variants(HDR10+, DV), I am 
> not against it. But it won't make the userspace be able to fill the HDR 
> metadata even the HDR data comes from the bitstream(likes SEI). We must 
> consider the case of Secure Video Path(Digital Right), the bitstream is 
> not accessible from (REE) userspace nor linux kernel, the downstream 
> must take what the upstream feed.

In that case, Secure Video Path is simply not possible to have (in
upstream). Keeping actual pixels restricted is one thing, and
understandable given they are the essence of the copyrighted material
that the providers want to keep inaccessible. Not being able to deliver
the metadata to userspace OTOH does not seem to have any justification
in comparison.


Thanks,
pq

> >>>> 2. Should I create a helper function for installing the SHMEM file as a fd?  
> >>> We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
> >>> so maybe that would be enough?
> >>>
> >>> [1]https://elixir.bootlin.com/linux/v6.5-rc7/source/drivers/dma-buf/udmabuf.c
> >>>      
> >> It is the kernel driver that allocate this buffer. For example, v4l2
> >> CAPTURE allocate a buffer for metadata when VIDIOC_REQBUFS.
> >> Or GBM give you a fd which is assigned with a surface.
> >>
> >> So we need a kernel interface.  
> >>> Best,
> >>> Tomasz
> >>>      
> >>>> --
> >>>> Hsia-Jun(Randy) Li  
> >   
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 8693 bytes --]

On Fri, 25 Aug 2023 15:56:18 +0800
Hsia-Jun Li <Randy.Li@synaptics.com> wrote:

> On 8/25/23 15:40, Pekka Paalanen wrote:
> > Subject:
> > Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers
> > From:
> > Pekka Paalanen <ppaalanen@gmail.com>
> > Date:
> > 8/25/23, 15:40
> > 
> > To:
> > Hsia-Jun Li <Randy.Li@synaptics.com>
> > CC:
> > Tomasz Figa <tfiga@chromium.org>, linux-mm@kvack.org, 
> > dri-devel@lists.freedesktop.org, Linux Media Mailing List 
> > <linux-media@vger.kernel.org>, hughd@google.com, 
> > akpm@linux-foundation.org, Simon Ser <contact@emersion.fr>, Hans Verkuil 
> > <hverkuil-cisco@xs4all.nl>, daniels@collabora.com, ayaka 
> > <ayaka@soulik.info>, linux-kernel@vger.kernel.org, Nicolas Dufresne 
> > <nicolas@ndufresne.ca>
> > 
> > 
> > On Wed, 23 Aug 2023 15:11:23 +0800
> > Hsia-Jun Li<Randy.Li@synaptics.com>  wrote:
> >   
> >> On 8/23/23 12:46, Tomasz Figa wrote:  
> >>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> >>>
> >>>
> >>> Hi Hsia-Jun,
> >>>
> >>> On Tue, Aug 22, 2023 at 8:14 PM Hsia-Jun Li<Randy.Li@synaptics.com>  wrote:  
> >>>> Hello
> >>>>
> >>>> I would like to introduce a usage of SHMEM slimier to DMA-buf, the major
> >>>> purpose of that is sharing metadata or just a pure container for cross
> >>>> drivers.
> >>>>
> >>>> We need to exchange some sort of metadata between drivers, likes dynamic
> >>>> HDR data between video4linux2 and DRM.  
> >>> If the metadata isn't too big, would it be enough to just have the
> >>> kernel copy_from_user() to a kernel buffer in the ioctl code?
> >>>      
> >>>> Or the graphics frame buffer is
> >>>> too complex to be described with plain plane's DMA-buf fd.
> >>>> An issue between DRM and V4L2 is that DRM could only support 4 planes
> >>>> while it is 8 for V4L2. It would be pretty hard for DRM to expend its
> >>>> interface to support that 4 more planes which would lead to revision of
> >>>> many standard likes Vulkan, EGL.  
> >>> Could you explain how a shmem buffer could be used to support frame
> >>> buffers with more than 4 planes?
> >>> If you are asking why we need this:  
> >> 1. metadata likes dynamic HDR tone data
> >> 2. DRM also challenges with this problem, let me quote what sima said:
> >> "another trick that we iirc used for afbc is that sometimes the planes
> >> have a fixed layout
> >> like nv12
> >> and so logically it's multiple planes, but you only need one plane slot
> >> to describe the buffer
> >> since I think afbc had the "we need more than 4 planes" issue too"
> >>
> >> Unfortunately, there are vendor pixel formats are not fixed layout.
> >>
> >> 3. Secure(REE, trusted video piepline) info.
> >>
> >> For how to assign such metadata data.
> >> In case with a drm fb_id, it is simple, we just add a drm plane property
> >> for it. The V4L2 interface is not flexible, we could only leave into
> >> CAPTURE request_fd as a control.  
> >>>> Also, there is no reason to consume a device's memory for the content
> >>>> that device can't read it, or wasting an entry of IOMMU for such data.  
> >>> That's right, but DMA-buf doesn't really imply any of those. DMA-buf
> >>> is just a kernel object with some backing memory. It's up to the
> >>> allocator to decide how the backing memory is allocated and up to the
> >>> importer on whether it would be mapped into an IOMMU.
> >>>      
> >> I just want to say it can't be allocated at the same place which was for
> >> those DMA bufs(graphics or compressed bitstream).
> >> This also could be answer for your first question, if we place this kind
> >> of buffer in a plane for DMABUF(importing) in V4L2, V4L2 core would try
> >> to prepare it, which could map it into IOMMU.
> >>  
> >>>> Usually, such a metadata would be the value should be written to a
> >>>> hardware's registers, a 4KiB page would be 1024 items of 32 bits registers.
> >>>>
> >>>> Still, I have some problems with SHMEM:
> >>>> 1. I don't want the userspace modify the context of the SHMEM allocated
> >>>> by the kernel, is there a way to do so?  
> >>> This is generally impossible without doing any of the two:
> >>> 1) copying the contents to an internal buffer not accessible to the
> >>> userspace, OR
> >>> 2) modifying any of the buffer mappings to read-only
> >>>
> >>> 2) can actually be more costly than 1) (depending on the architecture,
> >>> data size, etc.), so we shouldn't just discard the option of a simple
> >>> copy_from_user() in the ioctl.
> >>>      
> >> I don't want the userspace access it at all. So that won't be a problem.  
> > Hi,
> > 
> > if userspace cannot access things like an image's HDR metadata, then it
> > will be impossible for userspace to program KMS to have the correct
> > color pipeline, or to send intended HDR metadata to a video sink.
> > 
> > You cannot leave userspace out of HDR metadata handling, because quite
> > probably the V4L2 buffer is not the only thing on screen. That means
> > there must composition of multiple sources with different image
> > properties and metadata, which means it is no longer obvious what HDR
> > metadata should be sent to the video sink.
> > 
> > Even if it is a TV-like application rather than a windowed desktop, you
> > will still have other contents to composite: OSD (volume indicators,
> > channels indicators, program guide, ...), sub-titles, channel logos,
> > notifications... These components ideally should not change their
> > appearance arbitrarily with the main program content and metadata
> > changes. Either the metadata sent to the video sink is kept static and
> > the main program adapted on the fly, or main program metadata is sent
> > to the video sink and the additional content is adapted on the fly.
> > 
> > There is only one set of HDR metadata and one composited image that can
> > be sent to a video sink, so both must be chosen and produced correctly
> > at the source side. This cannot be done automatically inside KMS kernel
> > drivers.
> >   
> There may be some misunderstanding.
> Let suppose this HDR data is in a vendor specific format.
> Both upstream(decoder) and downstream(DRM) hardware devices are coming 
> from the same vendor.
> Then we just need to delivery the reference to this metadata buffer from 
> the upstream to downstream, both of drivers know how to handle it.
> 
> Despite the userspace, we just need to extend a wayland protocol that 
> making wayland compositor know how to receive the reference to the 
> metadata and set it to the DRM plane.

Hi,

FWIW, I don't think Wayland upstream (wayland-protocols) would be any
more welcoming to opaque vendor-specific proprietary data blobs than the
upstream Linux.

(To everyone not following #dri-devel: There was a fairly long IRC
discussion about how HDR metadata is an exceptionally poor use case
example for "hidden" ancillary data blobs, and then continuing to how a
mechanism to pass proprietary data blobs around would not be accepted
upstream:
https://oftc.irclog.whitequark.org/dri-devel/2023-08-25#1692950883-1692960435;
)

> If you want a common HDR formats for all HDR variants(HDR10+, DV), I am 
> not against it. But it won't make the userspace be able to fill the HDR 
> metadata even the HDR data comes from the bitstream(likes SEI). We must 
> consider the case of Secure Video Path(Digital Right), the bitstream is 
> not accessible from (REE) userspace nor linux kernel, the downstream 
> must take what the upstream feed.

In that case, Secure Video Path is simply not possible to have (in
upstream). Keeping actual pixels restricted is one thing, and
understandable given they are the essence of the copyrighted material
that the providers want to keep inaccessible. Not being able to deliver
the metadata to userspace OTOH does not seem to have any justification
in comparison.


Thanks,
pq

> >>>> 2. Should I create a helper function for installing the SHMEM file as a fd?  
> >>> We already have the udmabuf device [1] to turn a memfd into a DMA-buf,
> >>> so maybe that would be enough?
> >>>
> >>> [1]https://elixir.bootlin.com/linux/v6.5-rc7/source/drivers/dma-buf/udmabuf.c
> >>>      
> >> It is the kernel driver that allocate this buffer. For example, v4l2
> >> CAPTURE allocate a buffer for metadata when VIDIOC_REQBUFS.
> >> Or GBM give you a fd which is assigned with a surface.
> >>
> >> So we need a kernel interface.  
> >>> Best,
> >>> Tomasz
> >>>      
> >>>> --
> >>>> Hsia-Jun(Randy) Li  
> >   
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Daniel Stone]

Hi,

On Fri, 25 Aug 2023 at 08:56, Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
> On 8/25/23 15:40, Pekka Paalanen wrote:
> > if userspace cannot access things like an image's HDR metadata, then it
> > will be impossible for userspace to program KMS to have the correct
> > color pipeline, or to send intended HDR metadata to a video sink.
> >
> > You cannot leave userspace out of HDR metadata handling, because quite
> > probably the V4L2 buffer is not the only thing on screen. That means
> > there must composition of multiple sources with different image
> > properties and metadata, which means it is no longer obvious what HDR
> > metadata should be sent to the video sink.
> >
> > Even if it is a TV-like application rather than a windowed desktop, you
> > will still have other contents to composite: OSD (volume indicators,
> > channels indicators, program guide, ...), sub-titles, channel logos,
> > notifications... These components ideally should not change their
> > appearance arbitrarily with the main program content and metadata
> > changes. Either the metadata sent to the video sink is kept static and
> > the main program adapted on the fly, or main program metadata is sent
> > to the video sink and the additional content is adapted on the fly.
> >
> > There is only one set of HDR metadata and one composited image that can
> > be sent to a video sink, so both must be chosen and produced correctly
> > at the source side. This cannot be done automatically inside KMS kernel
> > drivers.
>
> There may be some misunderstanding.
> Let suppose this HDR data is in a vendor specific format.
> Both upstream(decoder) and downstream(DRM) hardware devices are coming
> from the same vendor.
> Then we just need to delivery the reference to this metadata buffer from
> the upstream to downstream, both of drivers know how to handle it.
>
> Despite the userspace, we just need to extend a wayland protocol that
> making wayland compositor know how to receive the reference to the
> metadata and set it to the DRM plane.
>
> If you want a common HDR formats for all HDR variants(HDR10+, DV), I am
> not against it. But it won't make the userspace be able to fill the HDR
> metadata even the HDR data comes from the bitstream(likes SEI). We must
> consider the case of Secure Video Path(Digital Right), the bitstream is
> not accessible from (REE) userspace nor linux kernel, the downstream
> must take what the upstream feed.

To summarise from IRC, so it's properly documented: the community will
not accept this.

The mechanism (shmem, dmabuf, copy_from_user, whatever) is _not_ the
problem. The problem is the concept.

There have been incredibly extensive discussions on this list about
colour management and HDR, summarised in documentation in the DRM
repository, as well as three talks at the last XDC. This design is the
result of discussion between many community participants - including
hardware vendors - who have all come up with a design which
prioritises transparency and explicit operation.

What you are suggesting is exactly the opposite of this. A design in
which opaque magic blobs are passed around and the kernel does unknown
things based on the contents of those blobs, contradicts this design.
(This is different to compression, where even if the format is
proprietary, the effect is well-understood - for the current
compression mechanisms, it is a lossless transform.)

The old Android Display Framework (ADF) was based around the same
design with blobs of opaque driver-specific data, where generic code -
either in the kernel or in userspace - could not understand the effect
of these blobs. This design was rejected, and we made a clear choice
to follow the DRM design principles instead.

Upstream will not accept any design which hides magic away. 'GKI is
hard', 'other vendors won't let us', etc, are not good enough reasons
to change our mind on this fundamental principle.

Cheers,
Daniel

Re: [RFC]: shmem fd for non-DMA buffer sharing cross drivers

[Daniel Stone]

Hi,

On Fri, 25 Aug 2023 at 08:56, Hsia-Jun Li <Randy.Li@synaptics.com> wrote:
> On 8/25/23 15:40, Pekka Paalanen wrote:
> > if userspace cannot access things like an image's HDR metadata, then it
> > will be impossible for userspace to program KMS to have the correct
> > color pipeline, or to send intended HDR metadata to a video sink.
> >
> > You cannot leave userspace out of HDR metadata handling, because quite
> > probably the V4L2 buffer is not the only thing on screen. That means
> > there must composition of multiple sources with different image
> > properties and metadata, which means it is no longer obvious what HDR
> > metadata should be sent to the video sink.
> >
> > Even if it is a TV-like application rather than a windowed desktop, you
> > will still have other contents to composite: OSD (volume indicators,
> > channels indicators, program guide, ...), sub-titles, channel logos,
> > notifications... These components ideally should not change their
> > appearance arbitrarily with the main program content and metadata
> > changes. Either the metadata sent to the video sink is kept static and
> > the main program adapted on the fly, or main program metadata is sent
> > to the video sink and the additional content is adapted on the fly.
> >
> > There is only one set of HDR metadata and one composited image that can
> > be sent to a video sink, so both must be chosen and produced correctly
> > at the source side. This cannot be done automatically inside KMS kernel
> > drivers.
>
> There may be some misunderstanding.
> Let suppose this HDR data is in a vendor specific format.
> Both upstream(decoder) and downstream(DRM) hardware devices are coming
> from the same vendor.
> Then we just need to delivery the reference to this metadata buffer from
> the upstream to downstream, both of drivers know how to handle it.
>
> Despite the userspace, we just need to extend a wayland protocol that
> making wayland compositor know how to receive the reference to the
> metadata and set it to the DRM plane.
>
> If you want a common HDR formats for all HDR variants(HDR10+, DV), I am
> not against it. But it won't make the userspace be able to fill the HDR
> metadata even the HDR data comes from the bitstream(likes SEI). We must
> consider the case of Secure Video Path(Digital Right), the bitstream is
> not accessible from (REE) userspace nor linux kernel, the downstream
> must take what the upstream feed.

To summarise from IRC, so it's properly documented: the community will
not accept this.

The mechanism (shmem, dmabuf, copy_from_user, whatever) is _not_ the
problem. The problem is the concept.

There have been incredibly extensive discussions on this list about
colour management and HDR, summarised in documentation in the DRM
repository, as well as three talks at the last XDC. This design is the
result of discussion between many community participants - including
hardware vendors - who have all come up with a design which
prioritises transparency and explicit operation.

What you are suggesting is exactly the opposite of this. A design in
which opaque magic blobs are passed around and the kernel does unknown
things based on the contents of those blobs, contradicts this design.
(This is different to compression, where even if the format is
proprietary, the effect is well-understood - for the current
compression mechanisms, it is a lossless transform.)

The old Android Display Framework (ADF) was based around the same
design with blobs of opaque driver-specific data, where generic code -
either in the kernel or in userspace - could not understand the effect
of these blobs. This design was rejected, and we made a clear choice
to follow the DRM design principles instead.

Upstream will not accept any design which hides magic away. 'GKI is
hard', 'other vendors won't let us', etc, are not good enough reasons
to change our mind on this fundamental principle.

Cheers,
Daniel