Re: [PATCH v7 9/9] docs/system: add basic virtio-gpu documentation

[Akihiko Odaki <akihiko.odaki@gmail.com>]

On 2023/08/18 8:47, Gurchetan Singh wrote:
> 
> 
> On Wed, Aug 16, 2023 at 10:28 PM Akihiko Odaki <akihiko.odaki@gmail.com 
> <mailto:akihiko.odaki@gmail.com>> wrote:
> 
>     On 2023/08/17 11:23, Gurchetan Singh wrote:
>      > From: Gurchetan Singh <gurchetansingh@chromium.org
>     <mailto:gurchetansingh@chromium.org>>
>      >
>      > This adds basic documentation for virtio-gpu.
>      >
>      > Suggested-by: Akihiko Odaki <akihiko.odaki@daynix.com
>     <mailto:akihiko.odaki@daynix.com>>
>      > Signed-off-by: Gurchetan Singh <gurchetansingh@chromium.org
>     <mailto:gurchetansingh@chromium.org>>
>      > Tested-by: Alyssa Ross <hi@alyssa.is <mailto:hi@alyssa.is>>
>      > Tested-by: Emmanouil Pitsidianakis
>     <manos.pitsidianakis@linaro.org <mailto:manos.pitsidianakis@linaro.org>>
>      > Reviewed-by: Emmanouil Pitsidianakis
>     <manos.pitsidianakis@linaro.org <mailto:manos.pitsidianakis@linaro.org>>
>      > ---
>      > v2: - Incorporated suggestions by Akihiko Odaki
>      >      - Listed the currently supported capset_names (Bernard)
>      >
>      > v3: - Incorporated suggestions by Akihiko Odaki and Alyssa Ross
>      >
>      > v4: - Incorporated suggestions by Akihiko Odaki
>      >
>      > v5: - Removed pci suffix from examples
>      >      - Verified that -device virtio-gpu-rutabaga works.  Strangely
>      >        enough, I don't remember changing anything, and I remember
>      >        it not working.  I did rebase to top of tree though.
>      >      - Fixed meson examples in crosvm docs
>      >
>      >   docs/system/device-emulation.rst   |   1 +
>      >   docs/system/devices/virtio-gpu.rst | 113
>     +++++++++++++++++++++++++++++
>      >   2 files changed, 114 insertions(+)
>      >   create mode 100644 docs/system/devices/virtio-gpu.rst
>      >
>      > diff --git a/docs/system/device-emulation.rst
>     b/docs/system/device-emulation.rst
>      > index 4491c4cbf7..1167f3a9f2 100644
>      > --- a/docs/system/device-emulation.rst
>      > +++ b/docs/system/device-emulation.rst
>      > @@ -91,6 +91,7 @@ Emulated Devices
>      >      devices/nvme.rst
>      >      devices/usb.rst
>      >      devices/vhost-user.rst
>      > +   devices/virtio-gpu.rst
>      >      devices/virtio-pmem.rst
>      >      devices/vhost-user-rng.rst
>      >      devices/canokey.rst
>      > diff --git a/docs/system/devices/virtio-gpu.rst
>     b/docs/system/devices/virtio-gpu.rst
>      > new file mode 100644
>      > index 0000000000..8c5c708272
>      > --- /dev/null
>      > +++ b/docs/system/devices/virtio-gpu.rst
>      > @@ -0,0 +1,113 @@
>      > +..
>      > +   SPDX-License-Identifier: GPL-2.0
>      > +
>      > +virtio-gpu
>      > +==========
>      > +
>      > +This document explains the setup and usage of the virtio-gpu device.
>      > +The virtio-gpu device paravirtualizes the GPU and display
>     controller.
>      > +
>      > +Linux kernel support
>      > +--------------------
>      > +
>      > +virtio-gpu requires a guest Linux kernel built with the
>      > +``CONFIG_DRM_VIRTIO_GPU`` option.
>      > +
>      > +QEMU virtio-gpu variants
>      > +------------------------
>      > +
>      > +QEMU virtio-gpu device variants come in the following form:
>      > +
>      > + * ``virtio-vga[-BACKEND]``
>      > + * ``virtio-gpu[-BACKEND][-INTERFACE]``
>      > + * ``vhost-user-vga``
>      > + * ``vhost-user-pci``
>      > +
>      > +**Backends:** QEMU provides a 2D virtio-gpu backend, and two
>     accelerated
>      > +backends: virglrenderer ('gl' device label) and rutabaga_gfx
>     ('rutabaga'
>      > +device label).  There is a vhost-user backend that runs the
>     graphics stack
>      > +in a separate process for improved isolation.
>      > +
>      > +**Interfaces:** QEMU further categorizes virtio-gpu device
>     variants based
>      > +on the interface exposed to the guest. The interfaces can be
>     classified
>      > +into VGA and non-VGA variants. The VGA ones are prefixed with
>     virtio-vga
>      > +or vhost-user-vga while the non-VGA ones are prefixed with
>     virtio-gpu or
>      > +vhost-user-gpu.
>      > +
>      > +The VGA ones always use the PCI interface, but for the non-VGA
>     ones, the
>      > +user can further pick between MMIO or PCI. For MMIO, the user
>     can suffix
>      > +the device name with -device, though vhost-user-gpu does not
>     support MMIO.
>      > +For PCI, the user can suffix it with -pci. Without these
>     suffixes, the
>      > +platform default will be chosen.
>      > +
>      > +virtio-gpu 2d
>      > +-------------
>      > +
>      > +The default 2D backend only performs 2D operations. The guest
>     needs to
>      > +employ a software renderer for 3D graphics.
>      > +
>      > +Typically, the software renderer is provided by `Mesa`_ or
>     `SwiftShader`_.
>      > +Mesa's implementations (LLVMpipe, Lavapipe and virgl below) work
>     out of box
>      > +on typical modern Linux distributions.
>      > +
>      > +.. parsed-literal::
>      > +    -device virtio-gpu
>      > +
>      > +.. _Mesa: https://www.mesa3d.org/ <https://www.mesa3d.org/>
>      > +.. _SwiftShader: https://github.com/google/swiftshader
>     <https://github.com/google/swiftshader>
>      > +
>      > +virtio-gpu virglrenderer
>      > +------------------------
>      > +
>      > +When using virgl accelerated graphics mode in the guest, OpenGL
>     API calls
>      > +are translated into an intermediate representation (see
>     `Gallium3D`_). The
>      > +intermediate representation is communicated to the host and the
>      > +`virglrenderer`_ library on the host translates the intermediate
>      > +representation back to OpenGL API calls.
>      > +
>      > +.. parsed-literal::
>      > +    -device virtio-gpu-gl
>      > +
>      > +.. _Gallium3D:
>     https://www.freedesktop.org/wiki/Software/gallium/
>     <https://www.freedesktop.org/wiki/Software/gallium/>
>      > +.. _virglrenderer:
>     https://gitlab.freedesktop.org/virgl/virglrenderer/
>     <https://gitlab.freedesktop.org/virgl/virglrenderer/>
>      > +
>      > +virtio-gpu rutabaga
>      > +-------------------
>      > +
>      > +virtio-gpu can also leverage `rutabaga_gfx`_ to provide `gfxstream`_
>      > +rendering and `Wayland display passthrough`_.  With the
>     gfxstream rendering
>      > +mode, GLES and Vulkan calls are forwarded to the host with minimal
>      > +modification.
>      > +
>      > +The crosvm book provides directions on how to build a
>     `gfxstream-enabled
>      > +rutabaga`_ and launch a `guest Wayland proxy`_.
>      > +
>      > +This device does require host blob support (``hostmem`` field
>     below). The
>      > +``hostmem`` field specifies the size of virtio-gpu host memory
>     window.
>      > +This is typically between 256M and 8G.
>      > +
>      > +At least one capset (see colon separated ``capset_names`` below)
>     must be
>      > +specified when starting the device.  The currently supported
>      > +``capset_names`` are ``gfxstream-vulkan`` and ``cross-domain``
>     on Linux
>      > +guests. For Android guests, ``gfxstream-gles`` is also supported.
>      > +
>      > +The device will try to auto-detect the wayland socket path if the
>      > +``cross-domain`` capset name is set.  The user may optionally
>     specify
>      > +``wayland_socket_path`` for non-standard paths.
>      > +
>      > +The ``wsi`` option can be set to ``surfaceless`` or ``headless``.
>      > +Surfaceless doesn't create a native window surface, but does
>     copy from the
>      > +render target to the Pixman buffer if a virtio-gpu 2D hypercall
>     is issued.
>      > +Headless is like surfaceless, but doesn't copy to the Pixman buffer.
>      > +Surfaceless is the default if ``wsi`` is not specified.
>      > +
>      > +.. parsed-literal::
>      > +    -device
>     virtio-gpu-rutabaga,capset_names=gfxstream-vulkan:cross-domain,
>      > +     
>       hostmem=8G,wayland_socket_path=/tmp/nonstandard/mock_wayland.sock,
>      > +       wsi=headless
>      > +
>      > +.. _rutabaga_gfx:
>     https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h <https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h>
>      > +.. _gfxstream:
>     https://android.googlesource.com/platform/hardware/google/gfxstream/
>     <https://android.googlesource.com/platform/hardware/google/gfxstream/>
>      > +.. _Wayland display passthrough:
>     https://www.youtube.com/watch?v=OZJiHMtIQ2M
>     <https://www.youtube.com/watch?v=OZJiHMtIQ2M>
>      > +.. _gfxstream-enabled rutabaga:
>     https://crosvm.dev/book/appendix/rutabaga_gfx.html
>     <https://crosvm.dev/book/appendix/rutabaga_gfx.html>
> 
>     You have different links for "rutabaga_gfx" and "gfxstream-enabled
>     rutabaga", but I think you only need one.
> 
> 
> Done.  Didn't resend the entire patch series (to avoid spamming list), 
> just did "in-reply-to".  The change is also available at:
> 
> https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8 <https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8>

The patch series now looks good so I finally decided to try this patch 
series, but I couldn't get it work.

I noticed gfxstream has page size hardcoded as 4 KiB, which broke my 
setup on M2 MacBook Air (running Asahi Linux) that has 16 KiB page. You 
may add code to check host page size and to report an error if it's not 
4 KiB to virtio-gpu-rutabaga, but I think it's trivial to fix gfxstream 
to query page size at runtime as QEMU and Rutabaga does so I hope you to 
do so. For testing purpose, I have replaced it with 16 KiB on my setup.

I also found some bugs on QEMU side so I added comments to respective 
patches.

Below is the logs from my last attempt of running vkgears:

$ VK_LOADER_DEBUG=all demos/build/src/vulkan/vkgears
INFO:             Vulkan Loader Version 1.3.243
LAYER:            Searching for layer manifest files
LAYER:               In following locations:
LAYER:                  /var/home/person/.config/vulkan/implicit_layer.d
LAYER:                  /etc/xdg/vulkan/implicit_layer.d
LAYER:                  /etc/vulkan/implicit_layer.d
LAYER: 
/var/home/person/.local/share/vulkan/implicit_layer.d
LAYER: 
/var/home/person/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER: 
/var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                  /usr/local/share/vulkan/implicit_layer.d
LAYER:                  /usr/share/vulkan/implicit_layer.d
LAYER:               Found the following files:
LAYER: 
/usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
INFO:             Found manifest file 
/usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file 
version "1.0.0")
LAYER:            Searching for layer manifest files
LAYER:               In following locations:
LAYER:                  /var/home/person/.config/vulkan/explicit_layer.d
LAYER:                  /etc/xdg/vulkan/explicit_layer.d
LAYER:                  /etc/vulkan/explicit_layer.d
LAYER: 
/var/home/person/.local/share/vulkan/explicit_layer.d
LAYER: 
/var/home/person/.local/share/flatpak/exports/share/vulkan/explicit_layer.d
LAYER: 
/var/lib/flatpak/exports/share/vulkan/explicit_layer.d
LAYER:                  /usr/local/share/vulkan/explicit_layer.d
LAYER:                  /usr/share/vulkan/explicit_layer.d
LAYER:               Found no files
DRIVER:           Searching for driver manifest files
DRIVER:              In following locations:
DRIVER:                 /var/home/person/.config/vulkan/icd.d
DRIVER:                 /etc/xdg/vulkan/icd.d
DRIVER:                 /etc/vulkan/icd.d
DRIVER:                 /var/home/person/.local/share/vulkan/icd.d
DRIVER: 
/var/home/person/.local/share/flatpak/exports/share/vulkan/icd.d
DRIVER:                 /var/lib/flatpak/exports/share/vulkan/icd.d
DRIVER:                 /usr/local/share/vulkan/icd.d
DRIVER:                 /usr/share/vulkan/icd.d
DRIVER:              Found the following files:
DRIVER: 
/usr/local/share/vulkan/icd.d/cereal_icd.aarch64.json
DRIVER:                 /usr/share/vulkan/icd.d/broadcom_icd.aarch64.json
DRIVER:                 /usr/share/vulkan/icd.d/freedreno_icd.aarch64.json
DRIVER:                 /usr/share/vulkan/icd.d/lvp_icd.aarch64.json
DRIVER:                 /usr/share/vulkan/icd.d/panfrost_icd.aarch64.json
DRIVER:                 /usr/share/vulkan/icd.d/radeon_icd.aarch64.json
DRIVER:           Found ICD manifest file 
/usr/local/share/vulkan/icd.d/cereal_icd.aarch64.json, version "1.0.0"
DEBUG | DRIVER:   Searching for ICD drivers named 
/usr/local/lib64/libvulkan_cereal.so
[VirtGpuDevice.cpp(71)]
[prio 6] virtgpu backend not enabling 
VIRTGPU_PARAM_CREATE_GUEST_HANDLE[AndroidHealthMonitor.cpp(275)]
[prio 4] HealthMonitor disabled. Returning nullptrI0818 21:00:07.980257 
154451 IntelDrmDecoder.cpp:38] IntelDrmDecoder created for context 2
DRIVER:           Found ICD manifest file 
/usr/share/vulkan/icd.d/broadcom_icd.aarch64.json, version "1.0.0"
DEBUG | DRIVER:   Searching for ICD drivers named 
/usr/lib64/libvulkan_broadcom.so
DRIVER:           Found ICD manifest file 
/usr/share/vulkan/icd.d/freedreno_icd.aarch64.json, version "1.0.0"
DEBUG | DRIVER:   Searching for ICD drivers named 
/usr/lib64/libvulkan_freedreno.so
DRIVER:           Found ICD manifest file 
/usr/share/vulkan/icd.d/lvp_icd.aarch64.json, version "1.0.0"
DEBUG | DRIVER:   Searching for ICD drivers named 
/usr/lib64/libvulkan_lvp.so
DRIVER:           Found ICD manifest file 
/usr/share/vulkan/icd.d/panfrost_icd.aarch64.json, version "1.0.0"
DEBUG | DRIVER:   Searching for ICD drivers named 
/usr/lib64/libvulkan_panfrost.so
DRIVER:           Found ICD manifest file 
/usr/share/vulkan/icd.d/radeon_icd.aarch64.json, version "1.0.0"
DEBUG | DRIVER:   Searching for ICD drivers named 
/usr/lib64/libvulkan_radeon.so
DEBUG | LAYER:    Loading layer library libVkLayer_MESA_device_select.so
INFO | LAYER:     Insert instance layer "VK_LAYER_MESA_device_select" 
(libVkLayer_MESA_device_select.so)
LAYER:            vkCreateInstance layer callstack setup to:
LAYER:               <Application>
LAYER:                 ||
LAYER:               <Loader>
LAYER:                 ||
LAYER:               VK_LAYER_MESA_device_select
LAYER:                       Type: Implicit
LAYER:                           Disable Env Var:  NODEVICE_SELECT
LAYER:                       Manifest: 
/usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                       Library:  libVkLayer_MESA_device_select.so
LAYER:                 ||
LAYER:               <Drivers>
I0818 21:00:08.014896  154451 VkDecoderGlobalState.cpp:443] Creating 
Vulkan instance for app: vkgears
INFO | DRIVER:    linux_read_sorted_physical_devices:
INFO | DRIVER:         Original order:
INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:         Sorted order:
INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
INFO | DRIVER:    linux_read_sorted_physical_devices:
INFO | DRIVER:         Original order:
INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:         Sorted order:
INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
DEBUG | DRIVER:   Copying old device 0 into new device 0
DEBUG | DRIVER:   Copying old device 1 into new device 1
INFO | DRIVER:    linux_read_sorted_physical_devices:
INFO | DRIVER:         Original order:
INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:         Sorted order:
INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
DEBUG | DRIVER:   Copying old device 0 into new device 0
DEBUG | DRIVER:   Copying old device 1 into new device 1
INFO | DRIVER:    linux_read_sorted_physical_devices:
INFO | DRIVER:         Original order:
INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:         Sorted order:
INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
DEBUG | DRIVER:   Copying old device 0 into new device 0
DEBUG | DRIVER:   Copying old device 1 into new device 1
ERROR:            loader_validate_device_extensions: Device extension 
VK_KHR_swapchain not supported by selected physical device or enabled 
layers.
ERROR:            vkCreateDevice: Failed to validate extensions in list
ERROR:            vkGetDeviceQueue2: Invalid device 
[VUID-vkGetDeviceQueue2-device-parameter]
Aborted (core dumped)