Re: Corrupted display changing screen colour depth in qemu-system-ppc/MacOS

[Gerd Hoffmann <kraxel@redhat.com>]

On Thu, Jun 16, 2022 at 09:54:46AM +0100, Mark Cave-Ayland wrote:
> Hi all,
> 
> Howard pointed me off-list to a corrupted screen display issue experienced
> when changing the screen colour depth in MacOS under qemu-system-ppc. I was
> able to reproduce it here, and noticed from the output that the issue was
> likely due to the host display not updating its depth accordingly.
> 
> After it was confirmed to me that this was working in QEMU 6.2, I was able
> to eventually bisect the problem down to this commit:
> 
> 
> cb8962c146b2633a4b04562281de9b2703bba849 is the first bad commit
> commit cb8962c146b2633a4b04562281de9b2703bba849
> Author: Marc-André Lureau <marcandre.lureau@redhat.com>
> Date:   Tue Feb 15 00:13:37 2022 +0400
> 
>     ui: do not create a surface when resizing a GL scanout
> 
>     qemu_console_resize() will create a blank surface and replace the
>     current scanout with it if called while the current scanout is
>     GL (texture or dmabuf).
> 
>     This is not only very costly, but also can produce glitches on the
>     display/listener side.
> 
>     Instead, compare the current console size with the fitting console
>     functions, which also works when the scanout is GL.
> 
>     Note: there might be still an unnecessary surface creation on calling
>     qemu_console_resize() when the size is actually changing, but display
>     backends currently rely on DisplaySurface details during
>     dpy_gfx_switch() to handle various resize aspects. We would need more
>     refactoring to handle resize without DisplaySurface, this is left for a
>     future improvement.
> 
>     Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
>     Message-Id: <20220214201337.1814787-4-marcandre.lureau@redhat.com>
>     Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> 
>  ui/console.c | 7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
> 
> 
> Some more background: the screen in qemu-system-ppc's MacOS is controlled
> via a custom driver written by Ben which uses the Bochs VBE registers to
> change the screen width/height/depth. The code used to do this can be found
> at https://gitlab.com/qemu-project/QemuMacDrivers/-/blob/master/QemuVGADriver/src/QemuVga.c#L354.
> 
> Looking at the changes in cb8962c146 my guess would be that either the
> updated check in qemu_console_resize() should also check to see if the
> surface depth is unchanged before exiting early, or that there is an extra
> update required in the VGA device when changing just the screen colour depth
> by itself.

Can you try ditch the QEMU_ALLOCATED_FLAG check added by the commit?

Which depth changes triggers this?  Going from direct color to a
paletted mode?

take care,
  Gerd