Re: [PATH 0/4] [RFC] Support virtual DRM

Re: [PATH 0/4] [RFC] Support virtual DRM

[Esaki Tomohito]

Hi, Thomas
Thank you for reply.

On 2021/06/21 16:10, Thomas Zimmermann wrote:
> Hi
> 
> Am 21.06.21 um 08:27 schrieb Tomohito Esaki:
>> Virtual DRM splits the overlay planes of a display controller into
>> multiple
>> virtual devices to allow each plane to be accessed by each process.
>>
>> This makes it possible to overlay images output from multiple
>> processes on a
>> display. For example, one process displays the camera image without
>> compositor
>> while another process overlays the UI.
> 
> I briefly looked over your patches. I didn't understand how this is
> different to the functionality of a compositor? Shouldn't this be solved
> in userspace?

I think when latency is important (e.g., AR, VR, for displaying camera
images in IVI systems), there may be use cases where the compositor
cannot be used.
Normally, when the image is passed through the compositor, it is
displayed after 2 VSYNC at most, because the compositor combines the
image with VSYNC synchronization. On the other hand, if we use vDRM, the
image will be displayed at the next VSYNC, so it will be displayed after
1 VSYNC at most.

Also, since the compositor is a single point of failure, we may not want
to make it dependent on it.

Best regards
Tomohito Esaki

Re: [PATH 0/4] [RFC] Support virtual DRM

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 1463 bytes --]

On Tue, 22 Jun 2021 13:02:59 +0900
Esaki Tomohito <etom@igel.co.jp> wrote:

> Hi, Thomas
> Thank you for reply.
> 
> On 2021/06/21 16:10, Thomas Zimmermann wrote:
> > Hi
> > 
> > Am 21.06.21 um 08:27 schrieb Tomohito Esaki:  
> >> Virtual DRM splits the overlay planes of a display controller into
> >> multiple
> >> virtual devices to allow each plane to be accessed by each process.
> >>
> >> This makes it possible to overlay images output from multiple
> >> processes on a
> >> display. For example, one process displays the camera image without
> >> compositor
> >> while another process overlays the UI.  
> > 
> > I briefly looked over your patches. I didn't understand how this is
> > different to the functionality of a compositor? Shouldn't this be solved
> > in userspace?  
> 
> I think when latency is important (e.g., AR, VR, for displaying camera
> images in IVI systems), there may be use cases where the compositor
> cannot be used.

Hi,

> Normally, when the image is passed through the compositor, it is
> displayed after 2 VSYNC at most, because the compositor combines the
> image with VSYNC synchronization.

This is not a universal fact. You can write a Wayland compositor that
consistently reaches app-to-screen latency of less than one monitor
refresh cycle, while also using KMS planes.

I believe Weston succeeds in this already if you write the Wayland
application accordingly.


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATH 0/4] [RFC] Support virtual DRM

[Michel Dänzer]

On 2021-06-22 9:57 a.m., Pekka Paalanen wrote:
> On Tue, 22 Jun 2021 13:02:59 +0900
> Esaki Tomohito <etom@igel.co.jp> wrote:
> 
>> Hi, Thomas
>> Thank you for reply.
>>
>> On 2021/06/21 16:10, Thomas Zimmermann wrote:
>>> Hi
>>>
>>> Am 21.06.21 um 08:27 schrieb Tomohito Esaki:  
>>>> Virtual DRM splits the overlay planes of a display controller into
>>>> multiple
>>>> virtual devices to allow each plane to be accessed by each process.
>>>>
>>>> This makes it possible to overlay images output from multiple
>>>> processes on a
>>>> display. For example, one process displays the camera image without
>>>> compositor
>>>> while another process overlays the UI.  
>>>
>>> I briefly looked over your patches. I didn't understand how this is
>>> different to the functionality of a compositor? Shouldn't this be solved
>>> in userspace?  
>>
>> I think when latency is important (e.g., AR, VR, for displaying camera
>> images in IVI systems), there may be use cases where the compositor
>> cannot be used.
> 
> Hi,
> 
>> Normally, when the image is passed through the compositor, it is
>> displayed after 2 VSYNC at most, because the compositor combines the
>> image with VSYNC synchronization.
> 
> This is not a universal fact. You can write a Wayland compositor that
> consistently reaches app-to-screen latency of less than one monitor
> refresh cycle, while also using KMS planes.
> 
> I believe Weston succeeds in this already if you write the Wayland
> application accordingly.

For a specific example, https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/1620 allows app-to-screen latency as low as ~6 ms (including a fixed 2 ms buffer to avoid skipped frames). mutter doesn't use KMS planes yet, but if anything I'd expect that to help rather than hurt for latency (if the compositor doesn't need to draw anything).


-- 
Earthling Michel Dänzer               |               https://redhat.com
Libre software enthusiast             |             Mesa and X developer

Re: [PATH 0/4] [RFC] Support virtual DRM

[Esaki Tomohito]

On 2021/06/23 17:04, Michel Dänzer wrote:
> On 2021-06-22 9:57 a.m., Pekka Paalanen wrote:
>> On Tue, 22 Jun 2021 13:02:59 +0900
>> Esaki Tomohito <etom@igel.co.jp> wrote:
>>
>>> Hi, Thomas
>>> Thank you for reply.
>>>
>>> On 2021/06/21 16:10, Thomas Zimmermann wrote:
>>>> Hi
>>>>
>>>> Am 21.06.21 um 08:27 schrieb Tomohito Esaki:  
>>>>> Virtual DRM splits the overlay planes of a display controller into
>>>>> multiple
>>>>> virtual devices to allow each plane to be accessed by each process.
>>>>>
>>>>> This makes it possible to overlay images output from multiple
>>>>> processes on a
>>>>> display. For example, one process displays the camera image without
>>>>> compositor
>>>>> while another process overlays the UI.  
>>>>
>>>> I briefly looked over your patches. I didn't understand how this is
>>>> different to the functionality of a compositor? Shouldn't this be solved
>>>> in userspace?  
>>>
>>> I think when latency is important (e.g., AR, VR, for displaying camera
>>> images in IVI systems), there may be use cases where the compositor
>>> cannot be used.
>>
>> Hi,
>>
>>> Normally, when the image is passed through the compositor, it is
>>> displayed after 2 VSYNC at most, because the compositor combines the
>>> image with VSYNC synchronization.
>>
>> This is not a universal fact. You can write a Wayland compositor that
>> consistently reaches app-to-screen latency of less than one monitor
>> refresh cycle, while also using KMS planes.
>>
>> I believe Weston succeeds in this already if you write the Wayland
>> application accordingly.
> 
> For a specific example, https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/1620 allows app-to-screen latency as low as ~6 ms (including a fixed 2 ms buffer to avoid skipped frames). mutter doesn't use KMS planes yet, but if anything I'd expect that to help rather than hurt for latency (if the compositor doesn't need to draw anything).

Thank you for providing specific examples.

Best regards
Esaki

Re: [PATH 0/4] [RFC] Support virtual DRM

[Thomas Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 1854 bytes --]

Hi

Am 22.06.21 um 06:02 schrieb Esaki Tomohito:
> Hi, Thomas
> Thank you for reply.
> 
> On 2021/06/21 16:10, Thomas Zimmermann wrote:
>> Hi
>>
>> Am 21.06.21 um 08:27 schrieb Tomohito Esaki:
>>> Virtual DRM splits the overlay planes of a display controller into
>>> multiple
>>> virtual devices to allow each plane to be accessed by each process.
>>>
>>> This makes it possible to overlay images output from multiple
>>> processes on a
>>> display. For example, one process displays the camera image without
>>> compositor
>>> while another process overlays the UI.
>>
>> I briefly looked over your patches. I didn't understand how this is
>> different to the functionality of a compositor? Shouldn't this be solved
>> in userspace?
> 
> I think when latency is important (e.g., AR, VR, for displaying camera
> images in IVI systems), there may be use cases where the compositor
> cannot be used.
> Normally, when the image is passed through the compositor, it is
> displayed after 2 VSYNC at most, because the compositor combines the
> image with VSYNC synchronization. On the other hand, if we use vDRM, the
> image will be displayed at the next VSYNC, so it will be displayed after
> 1 VSYNC at most.

Other commenters already addressed these points.

> 
> Also, since the compositor is a single point of failure, we may not want
> to make it dependent on it.

The kernel is also a single point of failure.

TBH I don't think this feature should be merged until there's a clear 
use case that cannot be solved in userspace idiomatically.

Best regards
Thomas

> 
> Best regards
> Tomohito Esaki
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [PATH 0/4] [RFC] Support virtual DRM

[Esaki Tomohito]

Hi, Maxime
Thank you for reply.

On 2021/06/21 18:24, Maxime Ripard wrote:
> Hi,
> 
> On Mon, Jun 21, 2021 at 09:10:19AM +0200, Thomas Zimmermann wrote:
>> Am 21.06.21 um 08:27 schrieb Tomohito Esaki:
>>> Virtual DRM splits the overlay planes of a display controller into multiple
>>> virtual devices to allow each plane to be accessed by each process.
>>>
>>> This makes it possible to overlay images output from multiple processes on a
>>> display. For example, one process displays the camera image without compositor
>>> while another process overlays the UI.
>>
>> I briefly looked over your patches. I didn't understand how this is
>> different to the functionality of a compositor? Shouldn't this be solved in
>> userspace?
> 
> I think there could be a bunch of use-cases for something that could
> "steal" a plane without the compositor knowing.
> 
> Something I'd really like to work at some point for example is that the
> downstream RaspberryPi display driver has a visual clue when it's
> running too hot or is in over-current.
> 
> I don't think this is the right solution though. The DT binding makes it
> far too static, and if there's a compositor I'd assume it would want to
> know about it somehow (at least if it's from the userspace) ?
> 

I will reconsider the DT bindings.

We want to separate the resources from the master in units of planes,
so we proposed virtual DRM.
By separating the plane from the master and making it appear as
a virtual DRM devicein userland, the plane can be accessed from
userland using the general DRM API.
What do you think about this idea?

Best Regards
Tomohito Esaki

Re: [PATH 0/4] [RFC] Support virtual DRM

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 2405 bytes --]

On Tue, Jun 22, 2021 at 01:36:48PM +0900, Esaki Tomohito wrote:
> Hi, Maxime
> Thank you for reply.
> 
> On 2021/06/21 18:24, Maxime Ripard wrote:
> > Hi,
> > 
> > On Mon, Jun 21, 2021 at 09:10:19AM +0200, Thomas Zimmermann wrote:
> >> Am 21.06.21 um 08:27 schrieb Tomohito Esaki:
> >>> Virtual DRM splits the overlay planes of a display controller into multiple
> >>> virtual devices to allow each plane to be accessed by each process.
> >>>
> >>> This makes it possible to overlay images output from multiple processes on a
> >>> display. For example, one process displays the camera image without compositor
> >>> while another process overlays the UI.
> >>
> >> I briefly looked over your patches. I didn't understand how this is
> >> different to the functionality of a compositor? Shouldn't this be solved in
> >> userspace?
> > 
> > I think there could be a bunch of use-cases for something that could
> > "steal" a plane without the compositor knowing.
> > 
> > Something I'd really like to work at some point for example is that the
> > downstream RaspberryPi display driver has a visual clue when it's
> > running too hot or is in over-current.
> > 
> > I don't think this is the right solution though. The DT binding makes it
> > far too static, and if there's a compositor I'd assume it would want to
> > know about it somehow (at least if it's from the userspace) ?
> > 
> 
> I will reconsider the DT bindings.
> 
> We want to separate the resources from the master in units of planes,
> so we proposed virtual DRM.
> By separating the plane from the master and making it appear as
> a virtual DRM devicein userland, the plane can be accessed from
> userland using the general DRM API.
> What do you think about this idea?

I guess you'd need to detail a bit more what your use case is exactly,
and what issue you're trying to address.

Generally speaking, I'm not really sure how you can separate a KMS
driver from its planes.

Like, assuming that you have that super important application putting
the rear-end camera on the display: I'd assume you want the connector
and bridges to remain enabled? How are you going to synchronize with the
compositor if it wants to disable it, or change resolution?

Similarly, some features exposed on the connector, like bpc, might
affect the input format you want to have for your planes?

Maxime

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATH 0/4] [RFC] Support virtual DRM

[Esaki Tomohito]

Hi, Enrico Weigelt
Thank you for reply.

On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:
> On 21.06.21 08:27, Tomohito Esaki wrote:
> 
> Hi,
> 
>> Virtual DRM splits the overlay planes of a display controller into multiple
>> virtual devices to allow each plane to be accessed by each process.
>>
>> This makes it possible to overlay images output from multiple processes on a
>> display. For example, one process displays the camera image without compositor
>> while another process overlays the UI.
> 
> Are you attempting to create an simple in-kernel compositor ?

I think the basic idea is the same as DRMlease.
We want to separate the resources from the master in units of planes,
so we proposed virtual DRM.
I think the advantage of vDRM is that you can use general DRM APIs
in userland.

> I don't think that's not the way to go, at least not by touching each
> single display driver, and not hardcoding the planes in DT.

Thank you for comment. I will reconsider about DT.

> What's the actual use case you're doing that for ? Why not using some
> userland compositor ?

I think when latency is important (e.g., AR, VR, for displaying camera
images in IVI systems), there may be use cases where the compositor
cannot be used.
Normally, when the image is passed through the compositor, it is
displayed after 2 VSYNC at most, because the compositor combines the
image with VSYNC synchronization. On the other hand, if we use vDRM, the
image will be displayed at the next VSYNC, so it will be displayed after
1 VSYNC at most.

Also, since the compositor is a single point of failure, we may not want
to make it dependent on it.

Best regards
Tomohito Esaki

Re: [PATH 0/4] [RFC] Support virtual DRM

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 3567 bytes --]

On Tue, 22 Jun 2021 13:03:39 +0900
Esaki Tomohito <etom@igel.co.jp> wrote:

> Hi, Enrico Weigelt
> Thank you for reply.
> 
> On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:
> > On 21.06.21 08:27, Tomohito Esaki wrote:
> > 
> > Hi,
> >   
> >> Virtual DRM splits the overlay planes of a display controller into multiple
> >> virtual devices to allow each plane to be accessed by each process.
> >>
> >> This makes it possible to overlay images output from multiple processes on a
> >> display. For example, one process displays the camera image without compositor
> >> while another process overlays the UI.  
> > 
> > Are you attempting to create an simple in-kernel compositor ?  
> 
> I think the basic idea is the same as DRMlease.

Hi,

indeed. Why not use DRM leases instead?

> We want to separate the resources from the master in units of planes,
> so we proposed virtual DRM.
> I think the advantage of vDRM is that you can use general DRM APIs
> in userland.

You do that with DRM leases too.

> > I don't think that's not the way to go, at least not by touching each
> > single display driver, and not hardcoding the planes in DT.  
> 
> Thank you for comment. I will reconsider about DT.
> 
> > What's the actual use case you're doing that for ? Why not using some
> > userland compositor ?  
> 
> I think when latency is important (e.g., AR, VR, for displaying camera
> images in IVI systems), there may be use cases where the compositor
> cannot be used.
> Normally, when the image is passed through the compositor, it is
> displayed after 2 VSYNC at most, because the compositor combines the
> image with VSYNC synchronization. On the other hand, if we use vDRM, the
> image will be displayed at the next VSYNC, so it will be displayed after
> 1 VSYNC at most.

As I said in my other email, this is false in the general sense.

> Also, since the compositor is a single point of failure, we may not want
> to make it dependent on it.

This... I'm not quite sure I buy it. If any of all the programs using
virtual KMS crashes, you still lose some crucial components from your
display. Maybe that program, while crashing, uploads such a bad state
to its very own KMS plane, that it causes other KMS planes to
malfunction. Then you need to detect this situation and still restart
everything, not just the crashed program.

I would think a userspace compositor approach is actually more
reliable. You write the compositor to be extremely robust. Exactly
because the compositor is in control of the complete display device and
not just little pieces of it, it can see what is happening and it can
mitigate problems. If you have more unreliable components needing
access to display, make those clients to the compositor, so they can
crash and malfunction on their own without potentially killing the
whole display device. If you are as concerned about latency as XR
people are, then use DRM leases.

Also, what if your virtual KMS driver has a bug? Restarting the kernel
is much harder that restarting a userspace compositor that hands out
DRM leases.

The userspace compositor could even be such that it does nothing more
than handing out DRM leases. However, DRM leases have the problem that
there is no single entity responsible for keeping the display device
working, but that responsibility is split between several processes and
none of them sees the whole picture.


Btw. VKMS is an existing DRM driver, so your name choice is conflicting.


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATH 0/4] [RFC] Support virtual DRM

[Daniel Vetter]

On Tue, Jun 22, 2021 at 10:12 AM Pekka Paalanen <ppaalanen@gmail.com> wrote:
>
> On Tue, 22 Jun 2021 13:03:39 +0900
> Esaki Tomohito <etom@igel.co.jp> wrote:
>
> > Hi, Enrico Weigelt
> > Thank you for reply.
> >
> > On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:
> > > On 21.06.21 08:27, Tomohito Esaki wrote:
> > >
> > > Hi,
> > >
> > >> Virtual DRM splits the overlay planes of a display controller into multiple
> > >> virtual devices to allow each plane to be accessed by each process.
> > >>
> > >> This makes it possible to overlay images output from multiple processes on a
> > >> display. For example, one process displays the camera image without compositor
> > >> while another process overlays the UI.
> > >
> > > Are you attempting to create an simple in-kernel compositor ?
> >
> > I think the basic idea is the same as DRMlease.
>
> Hi,
>
> indeed. Why not use DRM leases instead?
>
> > We want to separate the resources from the master in units of planes,
> > so we proposed virtual DRM.
> > I think the advantage of vDRM is that you can use general DRM APIs
> > in userland.
>
> You do that with DRM leases too.
>
> > > I don't think that's not the way to go, at least not by touching each
> > > single display driver, and not hardcoding the planes in DT.
> >
> > Thank you for comment. I will reconsider about DT.
> >
> > > What's the actual use case you're doing that for ? Why not using some
> > > userland compositor ?
> >
> > I think when latency is important (e.g., AR, VR, for displaying camera
> > images in IVI systems), there may be use cases where the compositor
> > cannot be used.
> > Normally, when the image is passed through the compositor, it is
> > displayed after 2 VSYNC at most, because the compositor combines the
> > image with VSYNC synchronization. On the other hand, if we use vDRM, the
> > image will be displayed at the next VSYNC, so it will be displayed after
> > 1 VSYNC at most.
>
> As I said in my other email, this is false in the general sense.
>
> > Also, since the compositor is a single point of failure, we may not want
> > to make it dependent on it.
>
> This... I'm not quite sure I buy it. If any of all the programs using
> virtual KMS crashes, you still lose some crucial components from your
> display. Maybe that program, while crashing, uploads such a bad state
> to its very own KMS plane, that it causes other KMS planes to
> malfunction. Then you need to detect this situation and still restart
> everything, not just the crashed program.

This, a hundred times. At least in general it's impossible to
guarantee resource isolation between different parts of a kms device -
everything is shared at least in some driver in funny ways.

The only thing we try to guarantee is that if you keep flipping the
same plane with same pixel format, stride, offset, absolutely
everything except the memory block unchanged, then that's guaranteed
to work. Everything else is off the table.

This is why the drm-lease design ended up with revoke support, because
if something goes wrong a superior instance (the compositor, the
kernel can't decide that for userspace) needs to decide whom to shoot
and revoke their access.

> I would think a userspace compositor approach is actually more
> reliable. You write the compositor to be extremely robust. Exactly
> because the compositor is in control of the complete display device and
> not just little pieces of it, it can see what is happening and it can
> mitigate problems. If you have more unreliable components needing
> access to display, make those clients to the compositor, so they can
> crash and malfunction on their own without potentially killing the
> whole display device. If you are as concerned about latency as XR
> people are, then use DRM leases.
>
> Also, what if your virtual KMS driver has a bug? Restarting the kernel
> is much harder that restarting a userspace compositor that hands out
> DRM leases.
>
> The userspace compositor could even be such that it does nothing more
> than handing out DRM leases. However, DRM leases have the problem that
> there is no single entity responsible for keeping the display device
> working, but that responsibility is split between several processes and
> none of them sees the whole picture.

Yeah I think a compositor for this use-case, written in Rust and
heavily audited/proofed is probably a lot more reliable than cobbling
ill-defined kernel driver code on top of barely-defined hw semantics
in resource-sharing cases.

> Btw. VKMS is an existing DRM driver, so your name choice is conflicting.

Yeah that too :-)
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [PATH 0/4] [RFC] Support virtual DRM

[Esaki Tomohito]

Hi,
Thank you all for your comments.

On 2021/06/22 17:12, Pekka Paalanen wrote:
> On Tue, 22 Jun 2021 13:03:39 +0900
> Esaki Tomohito <etom@igel.co.jp> wrote:
> 
>> Hi, Enrico Weigelt
>> Thank you for reply.
>>
>> On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:
>>> On 21.06.21 08:27, Tomohito Esaki wrote:
>>>
>>> Hi,
>>>   
>>>> Virtual DRM splits the overlay planes of a display controller into multiple
>>>> virtual devices to allow each plane to be accessed by each process.
>>>>
>>>> This makes it possible to overlay images output from multiple processes on a
>>>> display. For example, one process displays the camera image without compositor
>>>> while another process overlays the UI.  
>>>
>>> Are you attempting to create an simple in-kernel compositor ?  
>>
>> I think the basic idea is the same as DRMlease.
> 
> Hi,
> 
> indeed. Why not use DRM leases instead?
> 

In this use case, I understand that this is not possible with DRM lease,
am I wrong?
I understand that it’s not possible to lease a plane and update planes
on the same output independently from different processes in current DRM
lease.

If this is correct, what do you think of adding support for plane leases
to the DRM lease to handle this case?

Thanks,
Tomohito Esaki

Re: [PATH 0/4] [RFC] Support virtual DRM

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 2434 bytes --]

On Wed, 23 Jun 2021 15:56:05 +0900
Esaki Tomohito <etom@igel.co.jp> wrote:

> Hi,
> Thank you all for your comments.
> 
> On 2021/06/22 17:12, Pekka Paalanen wrote:
> > On Tue, 22 Jun 2021 13:03:39 +0900
> > Esaki Tomohito <etom@igel.co.jp> wrote:
> >   
> >> Hi, Enrico Weigelt
> >> Thank you for reply.
> >>
> >> On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:  
> >>> On 21.06.21 08:27, Tomohito Esaki wrote:
> >>>
> >>> Hi,
> >>>     
> >>>> Virtual DRM splits the overlay planes of a display controller into multiple
> >>>> virtual devices to allow each plane to be accessed by each process.
> >>>>
> >>>> This makes it possible to overlay images output from multiple processes on a
> >>>> display. For example, one process displays the camera image without compositor
> >>>> while another process overlays the UI.    
> >>>
> >>> Are you attempting to create an simple in-kernel compositor ?    
> >>
> >> I think the basic idea is the same as DRMlease.  
> > 
> > Hi,
> > 
> > indeed. Why not use DRM leases instead?
> >   
> 
> In this use case, I understand that this is not possible with DRM lease,
> am I wrong?
> I understand that it’s not possible to lease a plane and update planes
> on the same output independently from different processes in current DRM
> lease.
> 
> If this is correct, what do you think of adding support for plane leases
> to the DRM lease to handle this case?

Hi,

I would love to see support added for leasing individual planes,
especially to replace the virtual DRM proposal which seems to be
eradicating everything that atomic modesetting and nuclear pageflip
have built over the many years.

However, please note that "on the same output independently" is
physically impossible. Semantically, the planes define what a CRTC
scans out, and the CRTC defines the scanout timings. Therefore it is not
possible to update individual planes independently, they will all
always share the timings of the CRTC.

That combined with KMS not allowing multiple updates to be queued at
the same time for the same CRTC (atomic commits and legacy pageflips
returning EBUSY) makes the plane updates very much inter-dependent.

If you want to avoid EBUSY and have planes update on the vblank you
intended, you really need a userspace compositor to pull everything
together *before* submitting anything to the kernel.


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATH 0/4] [RFC] Support virtual DRM

[Esaki Tomohito]

On 2021/06/23 17:39, Pekka Paalanen wrote:
> On Wed, 23 Jun 2021 15:56:05 +0900
> Esaki Tomohito <etom@igel.co.jp> wrote:
> 
>> Hi,
>> Thank you all for your comments.
>>
>> On 2021/06/22 17:12, Pekka Paalanen wrote:
>>> On Tue, 22 Jun 2021 13:03:39 +0900
>>> Esaki Tomohito <etom@igel.co.jp> wrote:
>>>   
>>>> Hi, Enrico Weigelt
>>>> Thank you for reply.
>>>>
>>>> On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:  
>>>>> On 21.06.21 08:27, Tomohito Esaki wrote:
>>>>>
>>>>> Hi,
>>>>>     
>>>>>> Virtual DRM splits the overlay planes of a display controller into multiple
>>>>>> virtual devices to allow each plane to be accessed by each process.
>>>>>>
>>>>>> This makes it possible to overlay images output from multiple processes on a
>>>>>> display. For example, one process displays the camera image without compositor
>>>>>> while another process overlays the UI.    
>>>>>
>>>>> Are you attempting to create an simple in-kernel compositor ?    
>>>>
>>>> I think the basic idea is the same as DRMlease.  
>>>
>>> Hi,
>>>
>>> indeed. Why not use DRM leases instead?
>>>   
>>
>> In this use case, I understand that this is not possible with DRM lease,
>> am I wrong?
>> I understand that it’s not possible to lease a plane and update planes
>> on the same output independently from different processes in current DRM
>> lease.
>>
>> If this is correct, what do you think of adding support for plane leases
>> to the DRM lease to handle this case?
> 
> Hi,
> 
> I would love to see support added for leasing individual planes,
> especially to replace the virtual DRM proposal which seems to be
> eradicating everything that atomic modesetting and nuclear pageflip
> have built over the many years.
> 
> However, please note that "on the same output independently" is
> physically impossible. Semantically, the planes define what a CRTC
> scans out, and the CRTC defines the scanout timings. Therefore it is not
> possible to update individual planes independently, they will all
> always share the timings of the CRTC.
> 
> That combined with KMS not allowing multiple updates to be queued at
> the same time for the same CRTC (atomic commits and legacy pageflips
> returning EBUSY) makes the plane updates very much inter-dependent.
> 
> If you want to avoid EBUSY and have planes update on the vblank you
> intended, you really need a userspace compositor to pull everything
> together *before* submitting anything to the kernel.

Hi,

Thank you for your comments and advice.
I will consider leasing a plane.

Thanks,
Esaki

Re: [PATH 0/4] [RFC] Support virtual DRM

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 3553 bytes --]

On Wed, 23 Jun 2021 18:22:47 +0900
Esaki Tomohito <etom@igel.co.jp> wrote:

> On 2021/06/23 17:39, Pekka Paalanen wrote:
> > On Wed, 23 Jun 2021 15:56:05 +0900
> > Esaki Tomohito <etom@igel.co.jp> wrote:
> >   
> >> Hi,
> >> Thank you all for your comments.
> >>
> >> On 2021/06/22 17:12, Pekka Paalanen wrote:  
> >>> On Tue, 22 Jun 2021 13:03:39 +0900
> >>> Esaki Tomohito <etom@igel.co.jp> wrote:
> >>>     
> >>>> Hi, Enrico Weigelt
> >>>> Thank you for reply.
> >>>>
> >>>> On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:    
> >>>>> On 21.06.21 08:27, Tomohito Esaki wrote:
> >>>>>
> >>>>> Hi,
> >>>>>       
> >>>>>> Virtual DRM splits the overlay planes of a display controller into multiple
> >>>>>> virtual devices to allow each plane to be accessed by each process.
> >>>>>>
> >>>>>> This makes it possible to overlay images output from multiple processes on a
> >>>>>> display. For example, one process displays the camera image without compositor
> >>>>>> while another process overlays the UI.      
> >>>>>
> >>>>> Are you attempting to create an simple in-kernel compositor ?      
> >>>>
> >>>> I think the basic idea is the same as DRMlease.    
> >>>
> >>> Hi,
> >>>
> >>> indeed. Why not use DRM leases instead?
> >>>     
> >>
> >> In this use case, I understand that this is not possible with DRM lease,
> >> am I wrong?
> >> I understand that it’s not possible to lease a plane and update planes
> >> on the same output independently from different processes in current DRM
> >> lease.
> >>
> >> If this is correct, what do you think of adding support for plane leases
> >> to the DRM lease to handle this case?  
> > 
> > Hi,
> > 
> > I would love to see support added for leasing individual planes,
> > especially to replace the virtual DRM proposal which seems to be
> > eradicating everything that atomic modesetting and nuclear pageflip
> > have built over the many years.
> > 
> > However, please note that "on the same output independently" is
> > physically impossible. Semantically, the planes define what a CRTC
> > scans out, and the CRTC defines the scanout timings. Therefore it is not
> > possible to update individual planes independently, they will all
> > always share the timings of the CRTC.
> > 
> > That combined with KMS not allowing multiple updates to be queued at
> > the same time for the same CRTC (atomic commits and legacy pageflips
> > returning EBUSY) makes the plane updates very much inter-dependent.
> > 
> > If you want to avoid EBUSY and have planes update on the vblank you
> > intended, you really need a userspace compositor to pull everything
> > together *before* submitting anything to the kernel.  
> 
> Hi,
> 
> Thank you for your comments and advice.
> I will consider leasing a plane.

Hi,

I wish you considered a userspace compositor first, once more, with
passion.

It does not need to be Weston, and it does not need to use Wayland.
Just a userspace daemon that owns the whole display device and somehow
talks to whatever else wants stuff on screen.

I have not seen any evidence that leasing individual planes would do
you any good. I can easily see it doing you harm. I'm only saying that
it would be better than the virtual DRM proposal if you absolutely have
to go there. Please, consider not going there at all.

"On the same output independently" is not possible for the very simple
reason that the pixel data needs to be streamed serially to a monitor.


Thanks,
pq

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATH 0/4] [RFC] Support virtual DRM

[Esaki Tomohito]

On 2021/06/23 20:41, Pekka Paalanen wrote:
> On Wed, 23 Jun 2021 18:22:47 +0900
> Esaki Tomohito <etom@igel.co.jp> wrote:
> 
>> On 2021/06/23 17:39, Pekka Paalanen wrote:
>>> On Wed, 23 Jun 2021 15:56:05 +0900
>>> Esaki Tomohito <etom@igel.co.jp> wrote:
>>>   
>>>> Hi,
>>>> Thank you all for your comments.
>>>>
>>>> On 2021/06/22 17:12, Pekka Paalanen wrote:  
>>>>> On Tue, 22 Jun 2021 13:03:39 +0900
>>>>> Esaki Tomohito <etom@igel.co.jp> wrote:
>>>>>     
>>>>>> Hi, Enrico Weigelt
>>>>>> Thank you for reply.
>>>>>>
>>>>>> On 2021/06/22 1:05, Enrico Weigelt, metux IT consult wrote:    
>>>>>>> On 21.06.21 08:27, Tomohito Esaki wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>       
>>>>>>>> Virtual DRM splits the overlay planes of a display controller into multiple
>>>>>>>> virtual devices to allow each plane to be accessed by each process.
>>>>>>>>
>>>>>>>> This makes it possible to overlay images output from multiple processes on a
>>>>>>>> display. For example, one process displays the camera image without compositor
>>>>>>>> while another process overlays the UI.      
>>>>>>>
>>>>>>> Are you attempting to create an simple in-kernel compositor ?      
>>>>>>
>>>>>> I think the basic idea is the same as DRMlease.    
>>>>>
>>>>> Hi,
>>>>>
>>>>> indeed. Why not use DRM leases instead?
>>>>>     
>>>>
>>>> In this use case, I understand that this is not possible with DRM lease,
>>>> am I wrong?
>>>> I understand that it’s not possible to lease a plane and update planes
>>>> on the same output independently from different processes in current DRM
>>>> lease.
>>>>
>>>> If this is correct, what do you think of adding support for plane leases
>>>> to the DRM lease to handle this case?  
>>>
>>> Hi,
>>>
>>> I would love to see support added for leasing individual planes,
>>> especially to replace the virtual DRM proposal which seems to be
>>> eradicating everything that atomic modesetting and nuclear pageflip
>>> have built over the many years.
>>>
>>> However, please note that "on the same output independently" is
>>> physically impossible. Semantically, the planes define what a CRTC
>>> scans out, and the CRTC defines the scanout timings. Therefore it is not
>>> possible to update individual planes independently, they will all
>>> always share the timings of the CRTC.
>>>
>>> That combined with KMS not allowing multiple updates to be queued at
>>> the same time for the same CRTC (atomic commits and legacy pageflips
>>> returning EBUSY) makes the plane updates very much inter-dependent.
>>>
>>> If you want to avoid EBUSY and have planes update on the vblank you
>>> intended, you really need a userspace compositor to pull everything
>>> together *before* submitting anything to the kernel.  
>>
>> Hi,
>>
>> Thank you for your comments and advice.
>> I will consider leasing a plane.
> 
> Hi,
> 
> I wish you considered a userspace compositor first, once more, with
> passion.
> 
> It does not need to be Weston, and it does not need to use Wayland.
> Just a userspace daemon that owns the whole display device and somehow
> talks to whatever else wants stuff on screen.
> 
> I have not seen any evidence that leasing individual planes would do
> you any good. I can easily see it doing you harm. I'm only saying that
> it would be better than the virtual DRM proposal if you absolutely have
> to go there. Please, consider not going there at all.
> 
> "On the same output independently" is not possible for the very simple
> reason that the pixel data needs to be streamed serially to a monitor.
> 

Hi,

Thank you for your advice.
Once again, I'll consider a userspace compositor first.

Best regards
Esaki

[PATH 0/4] [RFC] Support virtual DRM

[Tomohito Esaki]

Hello
Sorry, there was a typo in the dri-devel mail address, so I've resend it.
---

Virtual DRM splits the overlay planes of a display controller into multiple
virtual devices to allow each plane to be accessed by each process.

This makes it possible to overlay images output from multiple processes on a
display. For example, one process displays the camera image without compositor
while another process overlays the UI.

Virtual DRM driver doesn’t directly control the display hardware and has no
access to the physical bus. Instead, the virtual DRM driver issues requests to
the standard DRM device driver (parent) when the hardware needs to be
controlled. The parent is modified to notify the virtual DRM driver of
interruptevents from the display hardware. Therefore, in order to use virtual
DRM, each DRM device driver needs to add code to support virutal DRM.

The only driver supported in this patch series is rcar-du. This patch series
is divided into multiple. The first patch adds vDRM feature to DRM, and the
second patch support vDRM for the rcar-du driver. The other patches add
documentation.

In particular, I would appreciate your advice on the following points:
* virtual DRM generalization
  I've only tested with rcar-du, is there anything I should consider to make
  virtual DRM work with other drivers?

* Integration to upstream
  I think it is a good idea to add virtual DRM to the DRM core functionality,
  but I would appreciate any suggestions on what needs to be improved for
  integration to upstream.

* dumb_create and fb_create callback
  I think that the dumb_create and fb_create callbacks need to be done by the
  parent, and it is preferable to use the parent's callbacks as they are.
  However, since the dumb buffer needs to be registered in the parent and
  the fb handle needs to be registered in the drm_file of the vDRM, the
  dumb_create callbacks from the parent driver cannot be used as is.
  Therefore, the current implementation of the dumb_create callback is
  workarround.
  What do you think is the best way to deal with this issue?


Tomohito Esaki (4):
  Add Virtual DRM device driver
  rcar-du: Add support virtual DRM device
  dt-bindings: display: Add virtual DRM
  doc-rst: Add virtual DRM documentation

 .../devicetree/bindings/display/vdrm.yaml     |  67 ++
 Documentation/gpu/drivers.rst                 |   1 +
 Documentation/gpu/vdrm.rst                    |  51 ++
 drivers/gpu/drm/Kconfig                       |   7 +
 drivers/gpu/drm/Makefile                      |   1 +
 drivers/gpu/drm/rcar-du/Kconfig               |   4 +
 drivers/gpu/drm/rcar-du/Makefile              |   1 +
 drivers/gpu/drm/rcar-du/rcar_du_crtc.c        |  42 +
 drivers/gpu/drm/rcar-du/rcar_du_crtc.h        |  13 +
 drivers/gpu/drm/rcar-du/rcar_du_drv.c         |  13 +
 drivers/gpu/drm/rcar-du/rcar_du_drv.h         |   3 +
 drivers/gpu/drm/rcar-du/rcar_du_vdrm.c        | 191 ++++
 drivers/gpu/drm/rcar-du/rcar_du_vdrm.h        |  67 ++
 drivers/gpu/drm/rcar-du/rcar_du_vsp.c         |  22 +
 drivers/gpu/drm/rcar-du/rcar_du_vsp.h         |   1 +
 drivers/gpu/drm/vdrm/vdrm_api.h               |  68 ++
 drivers/gpu/drm/vdrm/vdrm_drv.c               | 859 ++++++++++++++++++
 drivers/gpu/drm/vdrm/vdrm_drv.h               |  80 ++
 18 files changed, 1491 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/display/vdrm.yaml
 create mode 100644 Documentation/gpu/vdrm.rst
 create mode 100644 drivers/gpu/drm/rcar-du/rcar_du_vdrm.c
 create mode 100644 drivers/gpu/drm/rcar-du/rcar_du_vdrm.h
 create mode 100644 drivers/gpu/drm/vdrm/vdrm_api.h
 create mode 100644 drivers/gpu/drm/vdrm/vdrm_drv.c
 create mode 100644 drivers/gpu/drm/vdrm/vdrm_drv.h

-- 
2.25.1

Re: [PATH 0/4] [RFC] Support virtual DRM

[Simon Ser]

On Monday, June 21st, 2021 at 08:43, Tomohito Esaki <etom@igel.co.jp> wrote:

> Virtual DRM splits the overlay planes of a display controller into multiple
> virtual devices to allow each plane to be accessed by each process.
> This makes it possible to overlay images output from multiple processes on a
> display. For example, one process displays the camera image without compositor
> while another process overlays the UI.

Updating the KMS state from multiple processes doesn't sound like a
good idea. This opens up synchronization and global device limits
issues.

Are you aware of DRM leasing?