About new backend for GPU compute ROCm in qemu

About new backend for GPU compute ROCm in qemu

[Huang, Honglei]

Hi Michael, Alex, Dmitry, Akihiko,

I'm bringing AMD GPU compute ROCm based on virtio. I posted a ROCm over 
virtio
implementation to virglrenderer nine months ago (MR !1568 [1]). The ROCm 
side has
been supportted by ROCm offical.

Current implementation is a virtio gpu context type capset handled inside
virglrenderer, sharing the display path. That's an awkward fit, many
compute GPUs have no display engine at all.

Beyond that, sharing the display path is increasingly painful:

   - Compute hammers the queues more than graphics, so sharing
     virtio gpu's single control queue with display/virgl causes contention
     and display stutter.
   - Compute contexts need far more blob / shared memory than a display one.
   - Maybe needs a wider ROCm / compute stack, cause the render model 
fits poorly:
     rocprofiler (PC sampling, SQTT/SPM, counters, high bandwidth streams)
     and ROCgdb (wave control, address watch, async exceptions an
     out of band channel that must not block display).
   - Events, faults and GPU reset/SMI are async and don't map onto fences.
   - All of this is hard to extend cleanly inside a display capset.

On the QEMU/host side, would something like this be OK? One step, two parts:

   - a dedicated headless virtio gpu instance for compute.
   - that instance served by a separate ROCm backend library loaded
     in-process by QEMU.

That reuses the existing pluggable backend model, a second virtio gpu + a
backend library. It doesn't add dedicated queues for debug/profiling 
currently.

Waiting for reply and  happy to share more detail. Thanks!

[1] https://gitlab.freedesktop.org/virgl/virglrenderer/-/merge_requests/1568

Regards,
Honglei

Re: About new backend for GPU compute ROCm in qemu

[Alex Bennée]

"Huang, Honglei" <honghuan@amd.com> writes:

> Hi Michael, Alex, Dmitry, Akihiko,
>
> I'm bringing AMD GPU compute ROCm based on virtio. I posted a ROCm
> over virtio
> implementation to virglrenderer nine months ago (MR !1568 [1]). The
> ROCm side has
> been supportted by ROCm offical.
>
> Current implementation is a virtio gpu context type capset handled inside
> virglrenderer, sharing the display path. That's an awkward fit, many
> compute GPUs have no display engine at all.
>
> Beyond that, sharing the display path is increasingly painful:
>
>   - Compute hammers the queues more than graphics, so sharing
>     virtio gpu's single control queue with display/virgl causes contention
>     and display stutter.

Is this just due to iteration? From a layman's point of view I'm curious
as to what the queues are doing.

>   - Compute contexts need far more blob / shared memory than a display
>   one.

I guess weights and context are long lived blobs compared to rendering assets?

>   - Maybe needs a wider ROCm / compute stack, cause the render model
>     fits poorly:
>     rocprofiler (PC sampling, SQTT/SPM, counters, high bandwidth streams)
>     and ROCgdb (wave control, address watch, async exceptions an
>     out of band channel that must not block display).
>   - Events, faults and GPU reset/SMI are async and don't map onto fences.
>   - All of this is hard to extend cleanly inside a display capset.
>
> On the QEMU/host side, would something like this be OK? One step, two parts:
>
>   - a dedicated headless virtio gpu instance for compute.

An oft-asked for VirtIO model is virtio-npu but I wonder if there is
enough commonality for a virtio-compute device with the same sort native
context type handling to deal with the those that want to have guests
targeting specific hardware rather than going through an abstraction
like Vulkan Computer or OpenGL CL.

>   - that instance served by a separate ROCm backend library loaded
>     in-process by QEMU.

Is this library a binary blob or open source? The last time I looked at
ROCm I had to give up as my AMD card was in an Aarch64 AVA machine.

> That reuses the existing pluggable backend model, a second virtio gpu + a
> backend library. It doesn't add dedicated queues for debug/profiling
> currently.
>
> Waiting for reply and  happy to share more detail. Thanks!
>
> [1] https://gitlab.freedesktop.org/virgl/virglrenderer/-/merge_requests/1568
>
> Regards,
> Honglei

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: About new backend for GPU compute ROCm in qemu

[Akihiko Odaki]

On 2026/08/17 12:19, Huang, Honglei wrote:
> 
> Hi Michael, Alex, Dmitry, Akihiko,

Hi Honglei,

> 
> I'm bringing AMD GPU compute ROCm based on virtio. I posted a ROCm over 
> virtio
> implementation to virglrenderer nine months ago (MR !1568 [1]). The ROCm 
> side has
> been supportted by ROCm offical.
> 
> Current implementation is a virtio gpu context type capset handled inside
> virglrenderer, sharing the display path. That's an awkward fit, many
> compute GPUs have no display engine at all.

I think "sharing the display path" conflates several layers and makes 
the problem difficult to assess. It would help to identify the concrete 
constraint behind "awkward fit."

End-to-end, there are four relevant layers:

1. Host GPU stack: hardware, host kernel, and host userspace
2. Paravirtualization stack: virglrenderer and QEMU
3. Host/guest interface: virtio and the capset-specific command stream
4. Guest GPU stack: guest kernel and guest userspace

Orthogonally, acceleration is separate from display and scanout. A
physical device may provide both, but acceleration does not require a
display engine. Linux likewise exposes render and compute interfaces
separately from modesetting. The userspace interface virglrenderer uses 
is messy; there is Vulkan, EGL, OpenGL, and now you are adding ROCm. But 
there is one thing I must note is that acceleration and display is 
decoupled, and acceleration does not require display.

At the protocol layer, context command buffers are carried by 
VIRTIO_GPU_CMD_SUBMIT_3D. Scanout uses separate core virtio-gpu 
commands, and VIRTIO_GPU_CMD_GET_DISPLAY_INFO may report no enabled 
displays. At the implementation layer, QEMU handles scanout 
presentation. virgl_cmd_set_scanout() obtains resource information 
through virgl_renderer_resource_get_info() or 
virgl_renderer_resource_get_info_ext(). That does not make scanout a 
virglrenderer-owned display path.

Therefore, if "sharing the display path" means sharing the same device, 
control queue, and QEMU execution context, that identifies a possible 
source of contention. If it means that capsets or virglrenderer are 
inherently tied to display, I do not think that is accurate. Vulkan 
compute is already used through Venus with libkrun [2], and VCL proposes 
OpenCL support through virglrenderer [3].
> Beyond that, sharing the display path is increasingly painful:
> 
>    - Compute hammers the queues more than graphics, so sharing
>      virtio gpu's single control queue with display/virgl causes contention
>      and display stutter.

All non-cursor commands do share one control queue, but a fence avoids 
serialization.

There may still be implementation-level contention, and it is not
necessarily specific to compute. A sufficiently busy graphics workload
could expose the same bottlenecks. Possible contributors in current QEMU
include:

a) qemu_console_hw_gl_block() blocks the entire queue when QEMU only
    needs to fence scanout commands.

b) virtio_gpu_virgl_unmap_resource_blob() may also block the entire
    queue just to delay one command.

c) QEMU dispatches the control queue and calls into virglrenderer from
    its main-loop thread along with display work and many other things.
    Venus's render server can offload renderer work, but control-queue
    dispatch remains in QEMU's main loop.

In any case, I think you need to do some experiments to track down the 
real cause. a) is easy to check: just comment out all 
qemu_console_hw_gl_block() calls; it may corrupt display but removes the 
blocking. b) can also be tested by leaking the mappings instead of 
blocking the whole queue. Using a different display device like qxl 
tells whether c) is causing contention.

>    - Compute contexts need far more blob / shared memory than a display 
> one.

It is not a problem by itself. Frequent mapping and unmapping might 
amplify the second issue above, but that needs to be measured.

>    - Maybe needs a wider ROCm / compute stack, cause the render model 
> fits poorly:
>      rocprofiler (PC sampling, SQTT/SPM, counters, high bandwidth streams)
>      and ROCgdb (wave control, address watch, async exceptions an
>      out of band channel that must not block display).

virglrenderer does not impose a particular render model. That's why 
Vulkan Compute just works with Venus.

>    - Events, faults and GPU reset/SMI are async and don't map onto fences.>    - All of this is hard to extend cleanly inside a display capset.
Capset is not about display but determines the protocol of the 
VIRTIO_GPU_CMD_SUBMIT_3D command stream. You have described events, 
faults and GPU reset/SMI are async don't map onto fences that may be 
associated with VIRTIO_GPU_CMD_SUBMIT_3D which is dictated by capset. An 
additional feature may be necessary, and it may or may not be dictated 
by capset. The other things are irrelevant with the protocol capset 
represents; they are either behavioral or about different commands.

> 
> On the QEMU/host side, would something like this be OK? One step, two 
> parts:
> 
>    - a dedicated headless virtio gpu instance for compute.

A second device would isolate its virtqueues and device-wide
renderer_blocked state. That may be useful if measurements show that
these are the bottlenecks, but it is not yet clear that they are or that
a second device is the appropriate solution.

>    - that instance served by a separate ROCm backend library loaded
>      in-process by QEMU.

First, I think we need to establish why ROCm cannot or should not remain
in virglrenderer. The virglrenderer, Venus, and VCL maintainers are
likely better placed to advise on that boundary. Once the protocol
requirements and performance measurements are clear, we can assess the
appropriate QEMU integration.

[2] 
https://developers.redhat.com/articles/2025/06/05/how-we-improved-ai-inference-macos-podman-containers
[3] 
https://www.qualcomm.com/developer/blog/2024/10/vcl-virtio-gpu-opencl-driver

Regards,
Akihiko Odaki

> 
> That reuses the existing pluggable backend model, a second virtio gpu + a
> backend library. It doesn't add dedicated queues for debug/profiling 
> currently.
> 
> Waiting for reply and  happy to share more detail. Thanks!
> 
> [1] https://gitlab.freedesktop.org/virgl/virglrenderer/-/ 
> merge_requests/1568
> 
> Regards,
> Honglei