RFC: page-flip with damage?

RFC: page-flip with damage?

[Thomas Hellstrom]

Hi, list!

Page flips, while efficient on real hardware, aren't that efficient in 
other situations, like for virtual devices with local, or even worse, 
remote desktops.
We might ending up forwarding or encoding a couple of full frames worth 
of data instead of a small region at a cursor blink.

Now there is this extension EGL_KHR_swap_buffers_with_damage, and 
gnome-shell/wayland on KMS also has a damage region that it forwards all 
the way down to the function where page-flip is called.

So I'd like to start looking at page-flips with damage, meaning that the 
damage is an optional hint to the device about what part of the contents 
is actually updated. What would be the best way to implement this? I 
figure this can be done within the atomic context with a region attached 
to the plane state? Would we want to follow the EGL extension and 
forward an array of rects or for simplicity use a single bounding box? 
Both these options would be a great win.

Thanks,

Thomas

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: page-flip with damage?

[Rob Clark]

On Sun, Sep 24, 2017 at 1:41 PM, Thomas Hellstrom <thellstrom@vmware.com> wrote:
> Hi, list!
>
> Page flips, while efficient on real hardware, aren't that efficient in other
> situations, like for virtual devices with local, or even worse, remote
> desktops.
> We might ending up forwarding or encoding a couple of full frames worth of
> data instead of a small region at a cursor blink.
>
> Now there is this extension EGL_KHR_swap_buffers_with_damage, and
> gnome-shell/wayland on KMS also has a damage region that it forwards all the
> way down to the function where page-flip is called.
>
> So I'd like to start looking at page-flips with damage, meaning that the
> damage is an optional hint to the device about what part of the contents is
> actually updated. What would be the best way to implement this? I figure
> this can be done within the atomic context with a region attached to the
> plane state? Would we want to follow the EGL extension and forward an array
> of rects or for simplicity use a single bounding box? Both these options
> would be a great win.
>

If we do list of rects, I'd suggest that perhaps userspace should be
encouraged to break things down into non-overlapping rects.. else
kernel side gets to do some complicated thinking to avoid
re-transmitting the parts that are covered by multiple rects.

This is also a pretty useful thing for DSI command-mode panels (and I
guess eDP-psr), so I think worth solving.

Not sure if anyone has hardware that lets you queue up multiple rects?
 IIRC last time I looked at this for DSI, you could feed the hw a
single rect at a time and then when you get an irq back from the hw
feed it the next rect.  So multiple rects seems more useful if there
are updates on very disjoint parts of the screen.

OTOH the existing dirty ioctl takes a list of clip-rects.. maybe that
is an argument for a list instead of single rect?

BR,
-R
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: page-flip with damage?

[Noralf Trønnes]

Den 24.09.2017 23.32, skrev Rob Clark:
> On Sun, Sep 24, 2017 at 1:41 PM, Thomas Hellstrom <thellstrom@vmware.com> wrote:
>> Hi, list!
>>
>> Page flips, while efficient on real hardware, aren't that efficient in other
>> situations, like for virtual devices with local, or even worse, remote
>> desktops.
>> We might ending up forwarding or encoding a couple of full frames worth of
>> data instead of a small region at a cursor blink.
>>
>> Now there is this extension EGL_KHR_swap_buffers_with_damage, and
>> gnome-shell/wayland on KMS also has a damage region that it forwards all the
>> way down to the function where page-flip is called.
>>
>> So I'd like to start looking at page-flips with damage, meaning that the
>> damage is an optional hint to the device about what part of the contents is
>> actually updated. What would be the best way to implement this? I figure
>> this can be done within the atomic context with a region attached to the
>> plane state? Would we want to follow the EGL extension and forward an array
>> of rects or for simplicity use a single bounding box? Both these options
>> would be a great win.
>>
> If we do list of rects, I'd suggest that perhaps userspace should be
> encouraged to break things down into non-overlapping rects.. else
> kernel side gets to do some complicated thinking to avoid
> re-transmitting the parts that are covered by multiple rects.
>
> This is also a pretty useful thing for DSI command-mode panels (and I
> guess eDP-psr), so I think worth solving.
>
> Not sure if anyone has hardware that lets you queue up multiple rects?
>   IIRC last time I looked at this for DSI, you could feed the hw a
> single rect at a time and then when you get an irq back from the hw
> feed it the next rect.  So multiple rects seems more useful if there
> are updates on very disjoint parts of the screen.
>
> OTOH the existing dirty ioctl takes a list of clip-rects.. maybe that
> is an argument for a list instead of single rect?

There's a TODO that Daniel Vetter came up with during the tinydrm
review that touches this:
https://dri.freedesktop.org/docs/drm/gpu/todo.html#better-manual-upload-support-for-atomic

If I remember correctly struct drm_rect was preferred in-kernel over
struct drm_clip_rect.

tinydrm is geared mainly towards MIPI DBI type panels, where the pixels
are pushed through the command (MIPI DCS) interface (mostly SPI).
This means that there are major savings in reducing the area to be flushed.
AFAIK X windows only sends one rectangle at a time in the DIRTYFB call.

Noralf.

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

RE: page-flip with damage?

[Michal Lukaszek]

Thomas Hellstrom wrote:

> Page flips, while efficient on real hardware, aren't that efficient in 
> other situations, like for virtual devices with local, or even worse, 
> So I'd like to start looking at page-flips with damage

This is funny, it's exactly what we pointed out during the lighting talk at XDC2017
https://youtu.be/R2XHZd4uXRI?t=6h2m50s

And VMware gets a mention, too.

:)

Join forces?

Thanks,
Michal
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

RE: page-flip with damage?

[Deepak Singh Rawat]

Hi Michal,

Are you currently working on this feature. I started to look into page flip with damage for VMware drivers. 

Thanks,
Deepak

> -----Original Message-----
> From: dri-devel [mailto:dri-devel-bounces@lists.freedesktop.org] On Behalf
> Of Michal Lukaszek
> Sent: Tuesday, September 26, 2017 12:02 AM
> To: dri-devel@lists.freedesktop.org
> Subject: RE: page-flip with damage?
> 
> Thomas Hellstrom wrote:
> 
> 
> 
> > Page flips, while efficient on real hardware, aren't that efficient in
> 
> > other situations, like for virtual devices with local, or even worse,
> 
> > So I'd like to start looking at page-flips with damage
> 
> 
> 
> This is funny, it's exactly what we pointed out during the lighting talk at
> XDC2017
> 
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__youtu.be_R2XHZd4uXRI-3Ft-
> 3D6h2m50s&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=zOOG28inJK0762S
> xAf-cyZdStnd2NQpRu98lJP2iYGw&m=MeqFMzaER-
> gbFwJbxAovSVKR5y4mpUOeQkVPBC4Ixrs&s=kq0q9TAXS1cc-
> 8aAgqYcfeYyh6oLZOLs4NL9-ixdPzI&e=
> 
> 
> 
> And VMware gets a mention, too.
> 
> 
> 
> :)
> 
> 
> 
> Join forces?
> 
> 
> 
> Thanks,
> 
> Michal
> 
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lists.freedesktop.org_mailman_listinfo_dri-
> 2Ddevel&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=zOOG28inJK0762SxAf
> -cyZdStnd2NQpRu98lJP2iYGw&m=MeqFMzaER-
> gbFwJbxAovSVKR5y4mpUOeQkVPBC4Ixrs&s=ppNKRV_ldifYVpCbA-r-
> 6e1vc7_6Wu_KboAacOSdM30&e=
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: page-flip with damage?

[Daniel Vetter]

On Sun, Sep 24, 2017 at 07:41:45PM +0200, Thomas Hellstrom wrote:
> Hi, list!
> 
> Page flips, while efficient on real hardware, aren't that efficient in other
> situations, like for virtual devices with local, or even worse, remote
> desktops.
> We might ending up forwarding or encoding a couple of full frames worth of
> data instead of a small region at a cursor blink.
> 
> Now there is this extension EGL_KHR_swap_buffers_with_damage, and
> gnome-shell/wayland on KMS also has a damage region that it forwards all the
> way down to the function where page-flip is called.
> 
> So I'd like to start looking at page-flips with damage, meaning that the
> damage is an optional hint to the device about what part of the contents is
> actually updated. What would be the best way to implement this? I figure
> this can be done within the atomic context with a region attached to the
> plane state? Would we want to follow the EGL extension and forward an array
> of rects or for simplicity use a single bounding box? Both these options
> would be a great win.

So my rough plan for all this was:

- Add damage to drm_crtc_state, in screen coordinates. I think this is the
  most natural place for this, since it's what PSR/manual upload DSI want.
  It should also fit well for udl and tinydrm.  Virtual drivers like
  vmwgfx might need helpers to wrap this back to framebuffer rectangles,
  but that seems the odd case out - the framebuffer-based approach in the
  DIRTY_IOCTL forces most drivers to do a fancy lookup from fb to the
  crtc.

  Per-plane dirty rectangle seems to be an awkward in-betwen state, with
  all the confusion about whether it's pre/post scaled and how to best
  combine them. And then someone changes the background color of the crtc
  (or something like that), what happens then? I think pushing all that
  onto userspace is best, it can always ask for a complete flip if it's
  unclear whether it damaged the entire screen or not.

- Add a helper to the atomic core to implement fb_funcs->dirty on top of
  this new atomic state, so that drivers don't have to implement both the
  frontbuffer and the pageflip version of the _with_damage update. Just
  doing a flip (or an async one, if the driver supports that) should work
  well I think.

Wrt single dirty rect vs. rectlist: I'd opt for a single rect since that
makes the interface easier. Currently most drivers collapes the list
passed to fb_funcs->dirty to a single rect, so that seems good enough, and
a nice simplification of the interface (both uapi and driver).

Wrt enabling this on the legacy pageflip ioctl: I wouldn't bother, since
all drivers we reasonable care about are atomic already. One exception is
amdgpu, but that's going to be fixed soonish by merging the DC branch I
hope.

There's definitely lots of people who want to see this happen, that part
is for sure.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: page-flip with damage?

[Thomas Hellstrom]

On 09/26/2017 01:18 AM, Daniel Vetter wrote:
> On Sun, Sep 24, 2017 at 07:41:45PM +0200, Thomas Hellstrom wrote:
>> Hi, list!
>>
>> Page flips, while efficient on real hardware, aren't that efficient in other
>> situations, like for virtual devices with local, or even worse, remote
>> desktops.
>> We might ending up forwarding or encoding a couple of full frames worth of
>> data instead of a small region at a cursor blink.
>>
>> Now there is this extension EGL_KHR_swap_buffers_with_damage, and
>> gnome-shell/wayland on KMS also has a damage region that it forwards all the
>> way down to the function where page-flip is called.
>>
>> So I'd like to start looking at page-flips with damage, meaning that the
>> damage is an optional hint to the device about what part of the contents is
>> actually updated. What would be the best way to implement this? I figure
>> this can be done within the atomic context with a region attached to the
>> plane state? Would we want to follow the EGL extension and forward an array
>> of rects or for simplicity use a single bounding box? Both these options
>> would be a great win.
> So my rough plan for all this was:
>
> - Add damage to drm_crtc_state, in screen coordinates. I think this is the
>    most natural place for this, since it's what PSR/manual upload DSI want.
>    It should also fit well for udl and tinydrm.  Virtual drivers like
>    vmwgfx might need helpers to wrap this back to framebuffer rectangles,
>    but that seems the odd case out - the framebuffer-based approach in the
>    DIRTY_IOCTL forces most drivers to do a fancy lookup from fb to the
>    crtc.
>
>    Per-plane dirty rectangle seems to be an awkward in-betwen state, with
>    all the confusion about whether it's pre/post scaled and how to best
>    combine them. And then someone changes the background color of the crtc
>    (or something like that), what happens then? I think pushing all that
>    onto userspace is best, it can always ask for a complete flip if it's
>    unclear whether it damaged the entire screen or not.
>
> - Add a helper to the atomic core to implement fb_funcs->dirty on top of
>    this new atomic state, so that drivers don't have to implement both the
>    frontbuffer and the pageflip version of the _with_damage update. Just
>    doing a flip (or an async one, if the driver supports that) should work
>    well I think.

Sounds fine with me. I think the DIRTY_IOCTL was designed with 
user-space in mind. Typically
userspace has easier access to what part of the framebuffer was dirtied 
than to which parts
of all crtcs scanning out of the framebuffer were dirtied. But even 
vmwgfx eventually
translates this to crtc rects so I'm OK with that.

> Wrt single dirty rect vs. rectlist: I'd opt for a single rect since that
> makes the interface easier. Currently most drivers collapes the list
> passed to fb_funcs->dirty to a single rect, so that seems good enough, and
> a nice simplification of the interface (both uapi and driver).

I think multiple cliprects had its benefits for old X type rendering: 
Frontbuffer-type diagonal
lines benchmarked with x11perf and similar. Now that everybody's 
compositing that use-case is mostly gone, and as you say
most driver coalesce anyway. Even we to some extent, at least on newer 
"hardware" versions.

>
> Wrt enabling this on the legacy pageflip ioctl: I wouldn't bother, since
> all drivers we reasonable care about are atomic already. One exception is
> amdgpu, but that's going to be fixed soonish by merging the DC branch I
> hope.
>
> There's definitely lots of people who want to see this happen, that part
> is for sure.

Great.

> -Daniel


If anybody's already doing this or has code to share, please let me 
know. Or for that matter if somebody needs this stuff faster than 
vmwgfx's moderate development pace :).

/Thomas


_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: page-flip with damage?

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 2222 bytes --]

On Tue, 26 Sep 2017 09:07:45 -0700
Thomas Hellstrom <thellstrom@vmware.com> wrote:

> On 09/26/2017 01:18 AM, Daniel Vetter wrote:
> > On Sun, Sep 24, 2017 at 07:41:45PM +0200, Thomas Hellstrom wrote:  
> >> Hi, list!
> >>
> >> Page flips, while efficient on real hardware, aren't that efficient in other
> >> situations, like for virtual devices with local, or even worse, remote
> >> desktops.
> >> We might ending up forwarding or encoding a couple of full frames worth of
> >> data instead of a small region at a cursor blink.
> >>
> >> Now there is this extension EGL_KHR_swap_buffers_with_damage, and
> >> gnome-shell/wayland on KMS also has a damage region that it forwards all the
> >> way down to the function where page-flip is called.

> > Wrt single dirty rect vs. rectlist: I'd opt for a single rect since that
> > makes the interface easier. Currently most drivers collapes the list
> > passed to fb_funcs->dirty to a single rect, so that seems good enough, and
> > a nice simplification of the interface (both uapi and driver).  
> 
> I think multiple cliprects had its benefits for old X type rendering: 
> Frontbuffer-type diagonal
> lines benchmarked with x11perf and similar. Now that everybody's 
> compositing that use-case is mostly gone, and as you say
> most driver coalesce anyway. Even we to some extent, at least on newer 
> "hardware" versions.

Hi,

it still is awfully easy to create a pathological use case where
collapsing into a single rect adds a thousand-fold more pixels to the
damage than having more than one rectangle would allow: two tiny
animations at the opposite corners of a screen.

I'm thinking Wayland compositors here.

Simplifying damage regions while not crashing down to just one
rectangle is quite possible I believe. Let userspace do simplification
from hundreds down to a few rectangles, and then collapse into one
rectangle in the driver if absolutely necessary. That's what I'd hope.

It is totally fine to require non-overlapping rectangles. You could
even demand a specific layout of rectangles, e.g. the form Pixman
regions use. You can also put an arbitrary cap on how many rectangles
are allowed.


Thanks,
pq

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: page-flip with damage?

[Sean Paul]

On Thu, Oct 12, 2017 at 01:55:40PM +0300, Pekka Paalanen wrote:
> On Tue, 26 Sep 2017 09:07:45 -0700
> Thomas Hellstrom <thellstrom@vmware.com> wrote:
> 
> > On 09/26/2017 01:18 AM, Daniel Vetter wrote:
> > > On Sun, Sep 24, 2017 at 07:41:45PM +0200, Thomas Hellstrom wrote:  
> > >> Hi, list!
> > >>
> > >> Page flips, while efficient on real hardware, aren't that efficient in other
> > >> situations, like for virtual devices with local, or even worse, remote
> > >> desktops.
> > >> We might ending up forwarding or encoding a couple of full frames worth of
> > >> data instead of a small region at a cursor blink.
> > >>
> > >> Now there is this extension EGL_KHR_swap_buffers_with_damage, and
> > >> gnome-shell/wayland on KMS also has a damage region that it forwards all the
> > >> way down to the function where page-flip is called.
> 
> > > Wrt single dirty rect vs. rectlist: I'd opt for a single rect since that
> > > makes the interface easier. Currently most drivers collapes the list
> > > passed to fb_funcs->dirty to a single rect, so that seems good enough, and
> > > a nice simplification of the interface (both uapi and driver).  
> > 
> > I think multiple cliprects had its benefits for old X type rendering: 
> > Frontbuffer-type diagonal
> > lines benchmarked with x11perf and similar. Now that everybody's 
> > compositing that use-case is mostly gone, and as you say
> > most driver coalesce anyway. Even we to some extent, at least on newer 
> > "hardware" versions.
> 
> Hi,
> 
> it still is awfully easy to create a pathological use case where
> collapsing into a single rect adds a thousand-fold more pixels to the
> damage than having more than one rectangle would allow: two tiny
> animations at the opposite corners of a screen.
> 
> I'm thinking Wayland compositors here.
> 
> Simplifying damage regions while not crashing down to just one
> rectangle is quite possible I believe. Let userspace do simplification
> from hundreds down to a few rectangles, 

Sounds like you're making a case for overlay planes here :-)

Perhaps migrating the damage rect to plane state would make everyone happy. In
the case where the hardware or compositor only supports one plane, there is one
big rect. In the case where multiple planes are used, you rely on the compositor
to make plane assignments and pass down the individual damage regions per plane.

Sean


> and then collapse into one
> rectangle in the driver if absolutely necessary. That's what I'd hope.
> 
> It is totally fine to require non-overlapping rectangles. You could
> even demand a specific layout of rectangles, e.g. the form Pixman
> regions use. You can also put an arbitrary cap on how many rectangles
> are allowed.
> 
> 
> Thanks,
> pq



> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel


-- 
Sean Paul, Software Engineer, Google / Chromium OS
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: page-flip with damage?

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 4340 bytes --]

On Thu, 12 Oct 2017 10:51:22 -0400
Sean Paul <seanpaul@chromium.org> wrote:

> On Thu, Oct 12, 2017 at 01:55:40PM +0300, Pekka Paalanen wrote:
> > On Tue, 26 Sep 2017 09:07:45 -0700
> > Thomas Hellstrom <thellstrom@vmware.com> wrote:
> >   
> > > On 09/26/2017 01:18 AM, Daniel Vetter wrote:  
> > > > On Sun, Sep 24, 2017 at 07:41:45PM +0200, Thomas Hellstrom wrote:    
> > > >> Hi, list!
> > > >>
> > > >> Page flips, while efficient on real hardware, aren't that efficient in other
> > > >> situations, like for virtual devices with local, or even worse, remote
> > > >> desktops.
> > > >> We might ending up forwarding or encoding a couple of full frames worth of
> > > >> data instead of a small region at a cursor blink.
> > > >>
> > > >> Now there is this extension EGL_KHR_swap_buffers_with_damage, and
> > > >> gnome-shell/wayland on KMS also has a damage region that it forwards all the
> > > >> way down to the function where page-flip is called.  
> >   
> > > > Wrt single dirty rect vs. rectlist: I'd opt for a single rect since that
> > > > makes the interface easier. Currently most drivers collapes the list
> > > > passed to fb_funcs->dirty to a single rect, so that seems good enough, and
> > > > a nice simplification of the interface (both uapi and driver).    
> > > 
> > > I think multiple cliprects had its benefits for old X type rendering: 
> > > Frontbuffer-type diagonal
> > > lines benchmarked with x11perf and similar. Now that everybody's 
> > > compositing that use-case is mostly gone, and as you say
> > > most driver coalesce anyway. Even we to some extent, at least on newer 
> > > "hardware" versions.  
> > 
> > Hi,
> > 
> > it still is awfully easy to create a pathological use case where
> > collapsing into a single rect adds a thousand-fold more pixels to the
> > damage than having more than one rectangle would allow: two tiny
> > animations at the opposite corners of a screen.
> > 
> > I'm thinking Wayland compositors here.
> > 
> > Simplifying damage regions while not crashing down to just one
> > rectangle is quite possible I believe. Let userspace do simplification
> > from hundreds down to a few rectangles,   
> 
> Sounds like you're making a case for overlay planes here :-)
> 
> Perhaps migrating the damage rect to plane state would make everyone happy. In
> the case where the hardware or compositor only supports one plane, there is one
> big rect. In the case where multiple planes are used, you rely on the compositor
> to make plane assignments and pass down the individual damage regions per plane.

Hi Sean,

no, I wasn't really. I was thinking of a single big Wayland app window,
that has small changes roughly in opposite corners. Or two windows far
apart on the same output, updating at the same, and being composited
into a single framebuffer, e.g. because they are software-rendered.

The real world use case I have hit was with Xwayland with a fullscreen
X11 window getting these little updates scattered around. Xwayland
collapsing the damage into a single rect to be submitted to Weston
caused Weston to spend a lot of time in the pixman-renderer (software
compositing). Fixing it to submit detailed damage regions reduced
Weston's CPU usage significantly, down to a quarter or something in
that magnitude, if I recall correctly. This is an extreme example,
comparing a single rect to the original hundreds of rects. In practice,
one should have a damage region simplifier that optimizes the trade-off
between more pixels and more rects.

That example is not directly applicable to the question at hand, but I
do believe it extrapolates somewhat on the downsides of using only a
single rectangle per plane or crtc on cases where damage is a useful
hint.

After all, 2D-compositors should be able to provide detailed damage
regions if they are allowed to. Would this not also be in the interest
of power saving?


Thanks,
pq

> > and then collapse into one
> > rectangle in the driver if absolutely necessary. That's what I'd hope.
> > 
> > It is totally fine to require non-overlapping rectangles. You could
> > even demand a specific layout of rectangles, e.g. the form Pixman
> > regions use. You can also put an arbitrary cap on how many rectangles
> > are allowed.
> > 

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: page-flip with damage?

[Thomas Hellstrom]

On 09/26/2017 10:18 AM, Daniel Vetter wrote:
> On Sun, Sep 24, 2017 at 07:41:45PM +0200, Thomas Hellstrom wrote:
>> Hi, list!
>>
>> Page flips, while efficient on real hardware, aren't that efficient in other
>> situations, like for virtual devices with local, or even worse, remote
>> desktops.
>> We might ending up forwarding or encoding a couple of full frames worth of
>> data instead of a small region at a cursor blink.
>>
>> Now there is this extension EGL_KHR_swap_buffers_with_damage, and
>> gnome-shell/wayland on KMS also has a damage region that it forwards all the
>> way down to the function where page-flip is called.
>>
>> So I'd like to start looking at page-flips with damage, meaning that the
>> damage is an optional hint to the device about what part of the contents is
>> actually updated. What would be the best way to implement this? I figure
>> this can be done within the atomic context with a region attached to the
>> plane state? Would we want to follow the EGL extension and forward an array
>> of rects or for simplicity use a single bounding box? Both these options
>> would be a great win.
> So my rough plan for all this was:
>
> - Add damage to drm_crtc_state, in screen coordinates. I think this is the
>    most natural place for this, since it's what PSR/manual upload DSI want.
>    It should also fit well for udl and tinydrm.  Virtual drivers like
>    vmwgfx might need helpers to wrap this back to framebuffer rectangles,
>    but that seems the odd case out - the framebuffer-based approach in the
>    DIRTY_IOCTL forces most drivers to do a fancy lookup from fb to the
>    crtc.
>
>    Per-plane dirty rectangle seems to be an awkward in-betwen state, with
>    all the confusion about whether it's pre/post scaled and how to best
>    combine them. And then someone changes the background color of the crtc
>    (or something like that), what happens then? I think pushing all that
>    onto userspace is best, it can always ask for a complete flip if it's
>    unclear whether it damaged the entire screen or not.

Actually, after looking into this a bit in the context of remoting, I 
think I will have to disagree.

The most natural place for damage appears to be the drm_plane state, in 
plane fb coordinates.
The reason for this is that devices with hardware planes will want to 
have the coordinates in this way. Think cursor, Video overlay. Currently 
with vmwgfx and atomic we need to send the cursor image down the device 
pipeline on each cursor move which is pretty nasty.
Same thing holds for video overlay if we decide to move it to atomic. 
The overlay engine wants to know what part of the source image
has changed.
Damage is triggered by a content change and the change lives in the 
framebuffer, and this is easier on user-space as well.

Now for software plane compositing, you might be hitting the problems 
you describe above.
But if you're doing software plane compositing in your drm driver (which 
you probably aren't),
you'd have enough information anyway to handle both the odd case of 
background color change and any scaling if present (god forbid).

Thanks,
Thomas




_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: page-flip with damage?

[Daniel Vetter]

On Mon, Nov 27, 2017 at 09:03:22AM +0100, Thomas Hellstrom wrote:
> On 09/26/2017 10:18 AM, Daniel Vetter wrote:
> > On Sun, Sep 24, 2017 at 07:41:45PM +0200, Thomas Hellstrom wrote:
> > > Hi, list!
> > > 
> > > Page flips, while efficient on real hardware, aren't that efficient in other
> > > situations, like for virtual devices with local, or even worse, remote
> > > desktops.
> > > We might ending up forwarding or encoding a couple of full frames worth of
> > > data instead of a small region at a cursor blink.
> > > 
> > > Now there is this extension EGL_KHR_swap_buffers_with_damage, and
> > > gnome-shell/wayland on KMS also has a damage region that it forwards all the
> > > way down to the function where page-flip is called.
> > > 
> > > So I'd like to start looking at page-flips with damage, meaning that the
> > > damage is an optional hint to the device about what part of the contents is
> > > actually updated. What would be the best way to implement this? I figure
> > > this can be done within the atomic context with a region attached to the
> > > plane state? Would we want to follow the EGL extension and forward an array
> > > of rects or for simplicity use a single bounding box? Both these options
> > > would be a great win.
> > So my rough plan for all this was:
> > 
> > - Add damage to drm_crtc_state, in screen coordinates. I think this is the
> >    most natural place for this, since it's what PSR/manual upload DSI want.
> >    It should also fit well for udl and tinydrm.  Virtual drivers like
> >    vmwgfx might need helpers to wrap this back to framebuffer rectangles,
> >    but that seems the odd case out - the framebuffer-based approach in the
> >    DIRTY_IOCTL forces most drivers to do a fancy lookup from fb to the
> >    crtc.
> > 
> >    Per-plane dirty rectangle seems to be an awkward in-betwen state, with
> >    all the confusion about whether it's pre/post scaled and how to best
> >    combine them. And then someone changes the background color of the crtc
> >    (or something like that), what happens then? I think pushing all that
> >    onto userspace is best, it can always ask for a complete flip if it's
> >    unclear whether it damaged the entire screen or not.
> 
> Actually, after looking into this a bit in the context of remoting, I think
> I will have to disagree.
> 
> The most natural place for damage appears to be the drm_plane state, in
> plane fb coordinates.
> The reason for this is that devices with hardware planes will want to have
> the coordinates in this way. Think cursor, Video overlay. Currently with
> vmwgfx and atomic we need to send the cursor image down the device pipeline
> on each cursor move which is pretty nasty.
> Same thing holds for video overlay if we decide to move it to atomic. The
> overlay engine wants to know what part of the source image
> has changed.
> Damage is triggered by a content change and the change lives in the
> framebuffer, and this is easier on user-space as well.
> 
> Now for software plane compositing, you might be hitting the problems you
> describe above.
> But if you're doing software plane compositing in your drm driver (which you
> probably aren't),
> you'd have enough information anyway to handle both the odd case of
> background color change and any scaling if present (god forbid).

The one thing I'm somewhat worried about with per-plane damage rects is
correctly adding them up into an overall crtc rect for hw that needs it
(that's edp and manual upload dsi). There's some fixed point math involved
first to get from plane src to crtc dst coords, and then there's the fun
of figuring out which properties affect the entire screen or entire plane
(background, gamma tables, ...).

But I guess we can do that in a helper and share that code with all
drivers that need it.

I think we have 2 different kinds of drivers which would benefit from
this:

- drivers that composite on the "display"/remotely (like vmwgfx, or maybe
  also spice): Want damage in plane source coordinates. I think we could
  also group udl and spi/i2c-connected screens into this, since you can do
  the same buffer upload dance. Those drivers also might benefit from a
  list of rectangles (assuming the rectangles aren't too small). At least
  e.g. udl has a hw cursor which is also composited remotely.

- hw drivers where the selective upload happens after compositing, i.e.
  edp psr and dsi manual upload. Generally you get 1 rectangle, maybe 2 if
  they're well-seperated enough. Screen coordinates only, and for damage
  calculation we must take all compositing parameters into account.

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: page-flip with damage?

[Thomas Hellstrom]

On 11/27/2017 09:25 AM, Daniel Vetter wrote:
> On Mon, Nov 27, 2017 at 09:03:22AM +0100, Thomas Hellstrom wrote:
>> On 09/26/2017 10:18 AM, Daniel Vetter wrote:
>>> On Sun, Sep 24, 2017 at 07:41:45PM +0200, Thomas Hellstrom wrote:
>>>> Hi, list!
>>>>
>>>> Page flips, while efficient on real hardware, aren't that efficient in other
>>>> situations, like for virtual devices with local, or even worse, remote
>>>> desktops.
>>>> We might ending up forwarding or encoding a couple of full frames worth of
>>>> data instead of a small region at a cursor blink.
>>>>
>>>> Now there is this extension EGL_KHR_swap_buffers_with_damage, and
>>>> gnome-shell/wayland on KMS also has a damage region that it forwards all the
>>>> way down to the function where page-flip is called.
>>>>
>>>> So I'd like to start looking at page-flips with damage, meaning that the
>>>> damage is an optional hint to the device about what part of the contents is
>>>> actually updated. What would be the best way to implement this? I figure
>>>> this can be done within the atomic context with a region attached to the
>>>> plane state? Would we want to follow the EGL extension and forward an array
>>>> of rects or for simplicity use a single bounding box? Both these options
>>>> would be a great win.
>>> So my rough plan for all this was:
>>>
>>> - Add damage to drm_crtc_state, in screen coordinates. I think this is the
>>>     most natural place for this, since it's what PSR/manual upload DSI want.
>>>     It should also fit well for udl and tinydrm.  Virtual drivers like
>>>     vmwgfx might need helpers to wrap this back to framebuffer rectangles,
>>>     but that seems the odd case out - the framebuffer-based approach in the
>>>     DIRTY_IOCTL forces most drivers to do a fancy lookup from fb to the
>>>     crtc.
>>>
>>>     Per-plane dirty rectangle seems to be an awkward in-betwen state, with
>>>     all the confusion about whether it's pre/post scaled and how to best
>>>     combine them. And then someone changes the background color of the crtc
>>>     (or something like that), what happens then? I think pushing all that
>>>     onto userspace is best, it can always ask for a complete flip if it's
>>>     unclear whether it damaged the entire screen or not.
>> Actually, after looking into this a bit in the context of remoting, I think
>> I will have to disagree.
>>
>> The most natural place for damage appears to be the drm_plane state, in
>> plane fb coordinates.
>> The reason for this is that devices with hardware planes will want to have
>> the coordinates in this way. Think cursor, Video overlay. Currently with
>> vmwgfx and atomic we need to send the cursor image down the device pipeline
>> on each cursor move which is pretty nasty.
>> Same thing holds for video overlay if we decide to move it to atomic. The
>> overlay engine wants to know what part of the source image
>> has changed.
>> Damage is triggered by a content change and the change lives in the
>> framebuffer, and this is easier on user-space as well.
>>
>> Now for software plane compositing, you might be hitting the problems you
>> describe above.
>> But if you're doing software plane compositing in your drm driver (which you
>> probably aren't),
>> you'd have enough information anyway to handle both the odd case of
>> background color change and any scaling if present (god forbid).
> The one thing I'm somewhat worried about with per-plane damage rects is
> correctly adding them up into an overall crtc rect for hw that needs it
> (that's edp and manual upload dsi). There's some fixed point math involved
> first to get from plane src to crtc dst coords, and then there's the fun
> of figuring out which properties affect the entire screen or entire plane
> (background, gamma tables, ...).
>
> But I guess we can do that in a helper and share that code with all
> drivers that need it.
>
> I think we have 2 different kinds of drivers which would benefit from
> this:
>
> - drivers that composite on the "display"/remotely (like vmwgfx, or maybe
>    also spice): Want damage in plane source coordinates. I think we could
>    also group udl and spi/i2c-connected screens into this, since you can do
>    the same buffer upload dance. Those drivers also might benefit from a
>    list of rectangles (assuming the rectangles aren't too small). At least
>    e.g. udl has a hw cursor which is also composited remotely.
>
> - hw drivers where the selective upload happens after compositing, i.e.
>    edp psr and dsi manual upload. Generally you get 1 rectangle, maybe 2 if
>    they're well-seperated enough. Screen coordinates only, and for damage
>    calculation we must take all compositing parameters into account.

Are these latter type of drivers typically hw- or sw plane compositing?
In any case, I agree I think we would want a helper that turns plane 
source damage
into crtc damage. With the other way around, we can't really determine what
planes are damaged.

We could also perhaps submit damage per plane in plane crtc-side 
coordinates, It wouldn't
hurt video overlays much since they would probably do full size updates 
anyway.

/Thomas




>
> Cheers, Daniel


_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: page-flip with damage?

[Daniel Vetter]

On Mon, Nov 27, 2017 at 10:59:49AM +0100, Thomas Hellstrom wrote:
> On 11/27/2017 09:25 AM, Daniel Vetter wrote:
> > On Mon, Nov 27, 2017 at 09:03:22AM +0100, Thomas Hellstrom wrote:
> > > On 09/26/2017 10:18 AM, Daniel Vetter wrote:
> > > > On Sun, Sep 24, 2017 at 07:41:45PM +0200, Thomas Hellstrom wrote:
> > > > > Hi, list!
> > > > > 
> > > > > Page flips, while efficient on real hardware, aren't that efficient in other
> > > > > situations, like for virtual devices with local, or even worse, remote
> > > > > desktops.
> > > > > We might ending up forwarding or encoding a couple of full frames worth of
> > > > > data instead of a small region at a cursor blink.
> > > > > 
> > > > > Now there is this extension EGL_KHR_swap_buffers_with_damage, and
> > > > > gnome-shell/wayland on KMS also has a damage region that it forwards all the
> > > > > way down to the function where page-flip is called.
> > > > > 
> > > > > So I'd like to start looking at page-flips with damage, meaning that the
> > > > > damage is an optional hint to the device about what part of the contents is
> > > > > actually updated. What would be the best way to implement this? I figure
> > > > > this can be done within the atomic context with a region attached to the
> > > > > plane state? Would we want to follow the EGL extension and forward an array
> > > > > of rects or for simplicity use a single bounding box? Both these options
> > > > > would be a great win.
> > > > So my rough plan for all this was:
> > > > 
> > > > - Add damage to drm_crtc_state, in screen coordinates. I think this is the
> > > >     most natural place for this, since it's what PSR/manual upload DSI want.
> > > >     It should also fit well for udl and tinydrm.  Virtual drivers like
> > > >     vmwgfx might need helpers to wrap this back to framebuffer rectangles,
> > > >     but that seems the odd case out - the framebuffer-based approach in the
> > > >     DIRTY_IOCTL forces most drivers to do a fancy lookup from fb to the
> > > >     crtc.
> > > > 
> > > >     Per-plane dirty rectangle seems to be an awkward in-betwen state, with
> > > >     all the confusion about whether it's pre/post scaled and how to best
> > > >     combine them. And then someone changes the background color of the crtc
> > > >     (or something like that), what happens then? I think pushing all that
> > > >     onto userspace is best, it can always ask for a complete flip if it's
> > > >     unclear whether it damaged the entire screen or not.
> > > Actually, after looking into this a bit in the context of remoting, I think
> > > I will have to disagree.
> > > 
> > > The most natural place for damage appears to be the drm_plane state, in
> > > plane fb coordinates.
> > > The reason for this is that devices with hardware planes will want to have
> > > the coordinates in this way. Think cursor, Video overlay. Currently with
> > > vmwgfx and atomic we need to send the cursor image down the device pipeline
> > > on each cursor move which is pretty nasty.
> > > Same thing holds for video overlay if we decide to move it to atomic. The
> > > overlay engine wants to know what part of the source image
> > > has changed.
> > > Damage is triggered by a content change and the change lives in the
> > > framebuffer, and this is easier on user-space as well.
> > > 
> > > Now for software plane compositing, you might be hitting the problems you
> > > describe above.
> > > But if you're doing software plane compositing in your drm driver (which you
> > > probably aren't),
> > > you'd have enough information anyway to handle both the odd case of
> > > background color change and any scaling if present (god forbid).
> > The one thing I'm somewhat worried about with per-plane damage rects is
> > correctly adding them up into an overall crtc rect for hw that needs it
> > (that's edp and manual upload dsi). There's some fixed point math involved
> > first to get from plane src to crtc dst coords, and then there's the fun
> > of figuring out which properties affect the entire screen or entire plane
> > (background, gamma tables, ...).
> > 
> > But I guess we can do that in a helper and share that code with all
> > drivers that need it.
> > 
> > I think we have 2 different kinds of drivers which would benefit from
> > this:
> > 
> > - drivers that composite on the "display"/remotely (like vmwgfx, or maybe
> >    also spice): Want damage in plane source coordinates. I think we could
> >    also group udl and spi/i2c-connected screens into this, since you can do
> >    the same buffer upload dance. Those drivers also might benefit from a
> >    list of rectangles (assuming the rectangles aren't too small). At least
> >    e.g. udl has a hw cursor which is also composited remotely.
> > 
> > - hw drivers where the selective upload happens after compositing, i.e.
> >    edp psr and dsi manual upload. Generally you get 1 rectangle, maybe 2 if
> >    they're well-seperated enough. Screen coordinates only, and for damage
> >    calculation we must take all compositing parameters into account.
> 
> Are these latter type of drivers typically hw- or sw plane compositing?
> In any case, I agree I think we would want a helper that turns plane source
> damage
> into crtc damage. With the other way around, we can't really determine what
> planes are damaged.

All hw thus far. And if we ever get a vkms virtual driver modelling the
kms api, then we probably want the strictest interpretation of the damage
rects, i.e. at the plane level, and not converted to crtc damage first.

> We could also perhaps submit damage per plane in plane crtc-side
> coordinates, It wouldn't
> hurt video overlays much since they would probably do full size updates
> anyway.

I think once we have the helper function to convert to crtc damage it's
easy to write a few tests to make sure the sub-pixel handling is correct
when going from src to dst rects. And we have built-in tests now in drm
(just for drm_mm.c for now), so this should be simple to get going.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel