Re: About new backend for GPU compute ROCm in qemu

["Alex Bennée" <alex.bennee@linaro.org>]

"Huang, Honglei" <honghuan@amd.com> writes:

> Hi Michael, Alex, Dmitry, Akihiko,
>
> I'm bringing AMD GPU compute ROCm based on virtio. I posted a ROCm
> over virtio
> implementation to virglrenderer nine months ago (MR !1568 [1]). The
> ROCm side has
> been supportted by ROCm offical.
>
> Current implementation is a virtio gpu context type capset handled inside
> virglrenderer, sharing the display path. That's an awkward fit, many
> compute GPUs have no display engine at all.
>
> Beyond that, sharing the display path is increasingly painful:
>
>   - Compute hammers the queues more than graphics, so sharing
>     virtio gpu's single control queue with display/virgl causes contention
>     and display stutter.

Is this just due to iteration? From a layman's point of view I'm curious
as to what the queues are doing.

>   - Compute contexts need far more blob / shared memory than a display
>   one.

I guess weights and context are long lived blobs compared to rendering assets?

>   - Maybe needs a wider ROCm / compute stack, cause the render model
>     fits poorly:
>     rocprofiler (PC sampling, SQTT/SPM, counters, high bandwidth streams)
>     and ROCgdb (wave control, address watch, async exceptions an
>     out of band channel that must not block display).
>   - Events, faults and GPU reset/SMI are async and don't map onto fences.
>   - All of this is hard to extend cleanly inside a display capset.
>
> On the QEMU/host side, would something like this be OK? One step, two parts:
>
>   - a dedicated headless virtio gpu instance for compute.

An oft-asked for VirtIO model is virtio-npu but I wonder if there is
enough commonality for a virtio-compute device with the same sort native
context type handling to deal with the those that want to have guests
targeting specific hardware rather than going through an abstraction
like Vulkan Computer or OpenGL CL.

>   - that instance served by a separate ROCm backend library loaded
>     in-process by QEMU.

Is this library a binary blob or open source? The last time I looked at
ROCm I had to give up as my AMD card was in an Aarch64 AVA machine.

> That reuses the existing pluggable backend model, a second virtio gpu + a
> backend library. It doesn't add dedicated queues for debug/profiling
> currently.
>
> Waiting for reply and  happy to share more detail. Thanks!
>
> [1] https://gitlab.freedesktop.org/virgl/virglrenderer/-/merge_requests/1568
>
> Regards,
> Honglei

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro