Re: [RFC 0/6] RFC: hw/display/virtio-gpu: problems with coloured cursors

["Daniel P. Berrangé" <berrange@redhat.com>]

On Fri, Jan 24, 2025 at 11:00:33AM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> >    The cursor data virtio-gpu is receiving from the guest has
> >    had the alpha channel pre-multiplied into the RGB components.
> 
> The kernel driver simply passes through whatever it gets from userspace.
> 
> Not sure what userspace passes to the kernel, I suspect it is whatever
> typical GPUs can use unmodified as cursor sprite.
> 
> >    The cursor data virtio-gpu is receiving from the guest appears
> >    to be in BGRA8888 format, while GTK/VNC both expect the data to
> >    be in RGBA8888 format.
> 
> The format used by virtio-gpu is DRM_FORMAT_ARGB8888
> (VIRTIO_GPU_FORMAT_B8G8R8A8_UNORM) on little endian guests.  Byteswapped
> (DRM_FORMAT_BGRA8888 / VIRTIO_GPU_FORMAT_A8R8G8B8_UNORM) on bigendian
> guests.
> 
> > This series has patches to virtio-gpu which reverse the RGB
> > components and un-multiply the alpha channel.
> 
> I'd tend to simply pass the cursor format information to the ui and
> leave it to the ui to convert if needed, or just use the data as-is if
> possible (like SDL does).  Same approach we are using for the
> framebuffer data.
> 
> IIRC the cursor code predates the adoption of pixman in qemu, maybe it
> makes sense to redesign this around pixman images.

Yeah that all makes sense, but trying it I hit something odd. The
virtio_gpu_resource_create_2d *always* seems to report the format
as VIRTIO_GPU_FORMAT_B8G8R8X8_UNORM rather than
VIRTIO_GPU_FORMAT_B8G8R8A8_UNORM for the cursor resources.

IOW, the guest is incorrectly telling us there's no alpha channel
data, so if I honour the guest format, the cursor is filled in
black wherever there is supposed to be 100% alpha :-(

I can see a place in linux.git virtiogpu_gem.c that has harcoded
the DRM_FORMAT_HOST_XRGB8888 format for resources, which I guess
is where its going wrong :-(

I could try to workaround this in QEMU, and blindly assume that
guests always intend to have an alpha channel with cursors, as
it makes no sense to not do so.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|