qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* Help understanding allocation and mapping flow of virtio-gpu 3D resource blobs
@ 2024-02-04 11:06 Alex Bennée
  2024-02-04 11:18 ` Manos Pitsidianakis
  2024-02-04 14:52 ` Alex Bennée
  0 siblings, 2 replies; 3+ messages in thread
From: Alex Bennée @ 2024-02-04 11:06 UTC (permalink / raw)
  To: virglrenderer-devel-request, qemu-devel
  Cc: Gerd Hoffmann, Huang Rui, Antonio Caggiano, Dmitry Osipenko,
	Manos Pitsidianakis

Hi,

I'm trying to get an understanding of the blob allocation and mapping
flow for virtio-gpu for Vulkan and Rutabaga. Having gotten all the
various libraries setup I'm still seeing failures when running a TCG
guest (buildroot + latest glm, mesa, vkmark) with:

  ./qemu-system-aarch64 \
             -M virt -cpu cortex-a76 \
             -m 8192 \
             -object memory-backend-memfd,id=mem,size=8G,share=on \
             -serial mon:stdio \
             -kernel ~/lsrc/linux.git/builds/arm64.initramfs/arch/arm64/boot/Image \
             -append "console=ttyAMA0" \
             -device virtio-gpu-gl,context_init=true,blob=true,hostmem=4G \
             -display sdl,gl=on -d guest_errors,trace:virtio_gpu_cmd_res\*,trace:virtio_gpu_virgl_process_command -D debug.log

which shows up as detected in dmesg but not to vulkaninfo:

  [    0.644879] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)                
  [    0.648643] virtio-pci 0000:00:02.0: enabling device (0000 -> 0002)
  [    0.672391] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled                                                                                                      
  [    0.678071] Serial: AMBA driver                                                    
  [    0.682122] [drm] pci: virtio-gpu-pci detected at 0000:00:02.0           
  [    0.683249] [drm] Host memory window: 0x8000000000 +0x100000000
  [    0.683420] [drm] features: +virgl +edid +resource_blob +host_visible
  [    0.683470] [drm] features: +context_init                                                                                                                                 
  [    0.695695] [drm] number of scanouts: 1                                            
  [    0.695837] [drm] number of cap sets: 3                                            
  [    0.716173] [drm] cap set 0: id 1, max-version 1, max-size 308
  [    0.716499] [drm] cap set 1: id 2, max-version 2, max-size 1384          
  [    0.716686] [drm] cap set 2: id 4, max-version 0, max-size 160                                                                                                            
  [    0.726001] [drm] Initialized virtio_gpu 0.1.0 0 for 0000:00:02.0 on minor 0
  virgl_resource_create: err=0, res=2                                                   
  virgl_renderer_resource_attach_iov: 0x55b843c17a80/2                                  
  virgl_resource_attach_iov: pipe_resource: 0x55b8434da8f0                              
  vrend_pipe_resource_attach_iov: 0x43                                   

  ...

  # vulkaninfo --summary
  WARNING: [Loader Message] Code 0 : terminator_CreateInstance: Failed to CreateInstance in ICD 1.  Skipping ICD.
  error: XDG_RUNTIME_DIR is invalid or not set in the environment.
  ==========
  VULKANINFO
  ==========

  Vulkan Instance Version: 1.3.262


  Instance Extensions: count = 12
  -------------------------------
  VK_EXT_debug_report                    : extension revision 10
  VK_EXT_debug_utils                     : extension revision 2
  VK_KHR_device_group_creation           : extension revision 1
  VK_KHR_external_fence_capabilities     : extension revision 1
  VK_KHR_external_memory_capabilities    : extension revision 1
  VK_KHR_external_semaphore_capabilities : extension revision 1
  VK_KHR_get_physical_device_properties2 : extension revision 2
  VK_KHR_get_surface_capabilities2       : extension revision 1
  VK_KHR_portability_enumeration         : extension revision 1
  VK_KHR_surface                         : extension revision 25
  VK_KHR_surface_protected_capabilities  : extension revision 1
  VK_LUNARG_direct_driver_loading        : extension revision 1

  Instance Layers:
  ----------------

  Devices:
  ========
  GPU0:
          apiVersion         = 1.3.267
          driverVersion      = 0.0.1
          vendorID           = 0x10005
          deviceID           = 0x0000
          deviceType         = PHYSICAL_DEVICE_TYPE_CPU
          deviceName         = llvmpipe (LLVM 15.0.3, 128 bits)
          driverID           = DRIVER_ID_MESA_LLVMPIPE
          driverName         = llvmpipe
          driverInfo         = Mesa 23.3.2 (LLVM 15.0.3)
          conformanceVersion = 1.3.1.1
          deviceUUID         = 6d657361-3233-2e33-2e32-000000000000
          driverUUID         = 6c6c766d-7069-7065-5555-494400000000

With an older and more hacked up set of the blob patches I can get
vulkaninfo to work but I see multiple GPUs and vkmark falls over when
mapping stuff:

  # vulkaninfo --summary            
  render_state_create_resource: res_id = 5
  vkr_context_add_resource: res_id = 5
  vkr_context_import_resource_internal: res_id = 5
  virgl_resource_create: err=0, res=5                                                   
  render_state_create_resource: res_id = 6                                              
  vkr_context_add_resource: res_id = 6                                                  
  vkr_context_import_resource_internal: res_id = 6
  virgl_resource_create: err=0, res=6     
  error: XDG_RUNTIME_DIR is invalid or not set in the environment.
  ==========                                                                            
  VULKANINFO                                                                            
  ==========

  Vulkan Instance Version: 1.3.262  


  Instance Extensions: count = 12                                                       
  -------------------------------                                                       
  VK_EXT_debug_report                    : extension revision 10
  VK_EXT_debug_utils                     : extension revision 2
  VK_KHR_device_group_creation           : extension revision 1
  VK_KHR_external_fence_capabilities     : extension revision 1
  VK_KHR_external_memory_capabilities    : extension revision 1    
  VK_KHR_external_semaphore_capabilities : extension revision 1    
  VK_KHR_get_physical_device_properties2 : extension revision 2
  VK_KHR_get_surface_capabilities2       : extension revision 1
  VK_KHR_portability_enumeration         : extension revision 1
  VK_KHR_surface                         : extension revision 25
  VK_KHR_surface_protected_capabilities  : extension revision 1
  VK_LUNARG_direct_driver_loading        : extension revision 1
  VK_LUNARG_direct_driver_loading        : extension revision 1                                                                                                        [0/7869]

  Instance Layers:
  ----------------

  Devices:
  ========
  GPU0:
          apiVersion         = 1.3.230
          driverVersion      = 23.3.4
          vendorID           = 0x8086
          deviceID           = 0xa780
          deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
          deviceName         = Virtio-GPU Venus (Intel(R) Graphics (RPL-S))
          driverID           = DRIVER_ID_MESA_VENUS
          driverName         = venus
          driverInfo         = Mesa 23.3.4
          conformanceVersion = 1.3.0.0
          deviceUUID         = 29d2e940-a1a0-3054-0f9a-9f7dec52a084
          driverUUID         = b11fafe9-8706-9ab8-0f16-8b272cf893ca
  GPU1:
          apiVersion         = 1.2.0
          driverVersion      = 23.3.4
          vendorID           = 0x10005
          deviceID           = 0x0000
          deviceType         = PHYSICAL_DEVICE_TYPE_CPU
          deviceName         = Virtio-GPU Venus (llvmpipe (LLVM 15.0.6, 256 bits))
          driverID           = DRIVER_ID_MESA_VENUS
          driverName         = venus
          driverInfo         = Mesa 23.3.4
          conformanceVersion = 1.3.0.0
          deviceUUID         = 5fb5c03f-c537-f0fe-a7e6-9cd5866acb8d
          driverUUID         = b11fafe9-8706-9ab8-0f16-8b272cf893ca
  GPU2:
          apiVersion         = 1.3.267
          driverVersion      = 0.0.1
          vendorID           = 0x10005
          deviceID           = 0x0000
          deviceType         = PHYSICAL_DEVICE_TYPE_CPU
          deviceName         = llvmpipe (LLVM 15.0.3, 128 bits)
          driverID           = DRIVER_ID_MESA_LLVMPIPE
          driverName         = llvmpipe
          driverInfo         = Mesa 23.3.2 (LLVM 15.0.3)
          conformanceVersion = 1.3.1.1
          deviceUUID         = 6d657361-3233-2e33-2e32-000000000000
          driverUUID         = 6c6c766d-7069-7065-5555-494400000000
  render_state_destroy_resource: res_id = 5
  vkr_context_remove_resource: res_id = 5
  virgl_resource_destroy_func: res=5
  render_state_destroy_resource: res_id = 6
  vkr_context_remove_resource: res_id = 6
  virgl_resource_destroy_func: res=6

running vkmark gives:

  # vkmark --winsys kms
  render_state_create_resource: res_id = 7
  vkr_context_add_resource: res_id = 7
  vkr_context_import_resource_internal: res_id = 7
  virgl_resource_create: err=0, res=7
  render_state_create_resource: res_id = 8
  vkr_context_add_resource: res_id = 8
  vkr_context_import_resource_internal: res_id = 8
  virgl_resource_create: err=0, res=8
  virgl_resource_create: err=0, res=9
  virgl_renderer_resource_attach_iov: 0x55615acf7f40/9
  virgl_resource_attach_iov: pipe_resource: 0x55615acf7db0
  vrend_pipe_resource_attach_iov: 0x43

this bit does nothing as VREND_STORAGE_HOST_SYSTEM_MEMORY isn't set.

  virgl_resource_create: err=0, res=10
  virgl_renderer_resource_attach_iov: 0x55615ae569a0/10
  virgl_resource_attach_iov: pipe_resource: 0x55615a99ce20
  vrend_pipe_resource_attach_iov: 0x43
  Warning: KMSWindowSystem: Using VK_IMAGE_TILING_OPTIMAL for dmabuf with invalid modifier, but this is not guaranteed to work.
  vkr_dispatch_vkAllocateMemory: mem_index_type:0
  virgl_render_server[2889817]: vkr: failed to import resource: invalid res_id 9
  virgl_render_server[2889817]: vkr: vkAllocateMemory resulted in CS error
  virgl_render_server[2889817]: vkr: ring_submit_cmd: vn_dispatch_command failed
  MESA-VIRTIO: debug: vn_ring_submit abort on fatal
  render_state_destroy_resource: res_id = 7
  vkr_context_remove_resource: res_id = 7
  render_state_destroy_resource: res_id = 8
  vkr_context_remove_resource: res_id = 8
  virgl_resource_destroy_func: res=7
  virgl_resource_destroy_func: res=8
  virgl_resource_destroy_func: res=10
  virgl_resource_destroy_func: res=9
  Aborted

The debug printfs are throughout QEMU and virlgrenderer while trying to
work out what was going on. While the eventual aim is to enable this
stack on a ARM64 platform with Xen I wanted to make sure I understand
the blob resource flow first.

As I understand it we need resource blobs to hold 3D data for rendering
that is visible to the underlying GPU that will be doing the work. While
these blobs can be copied back and forth the most efficient way to do
this is to allocate HOST VISIBLE blobs that are appropriately placed
and aligned for the host GPU. For Vulkan there will also be the command
stream which will need to be translated from the Vulkan's portable
shader IR (as emitted by mesa on the guest) to the underlying shader
commands on the host system (via mesa on the host).

I can see the host visible region is created as a large chunk of the PCI
BAR space and is outside of the guests system RAM.

The guest creates a unique resource ID by submitting a
VIRTIO_GPU_CMD_RESOURCE_CREATE_BLOB command.

Q1: is the actual allocation done here? I assume this happens somewhere
in the depths of virglrenderer or is it the kernel DRM subsystem?

Q2: should the memory for this resource be visible to the host
userspace at this point? Where is the mapping into userspace done?

The guest submits a VIRTIO_GPU_CMD_RESOURCE_MAP_BLOB command with a
unique resource ID and requests it is mapped at an offset into the host
visible memory region.

Q3: Is all the mapping for host and guest handled by QEMU's memory_region
code?

Q4: Are there any differences between cards with VRAM and those with a
unified memory architecture (e.g. using system RAM)?

Finally should we expect to see any other resources
(RESOURCE_CREATE_2D/3D, TRANSFER_TO_HOST, ATTACH_BACKING) if we have
host visible blobs properly allocated and working?

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro


^ permalink raw reply	[flat|nested] 3+ messages in thread
end of thread, other threads:[~2024-02-04 14:52 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-02-04 11:06 Help understanding allocation and mapping flow of virtio-gpu 3D resource blobs Alex Bennée
2024-02-04 11:18 ` Manos Pitsidianakis
2024-02-04 14:52 ` Alex Bennée

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).