PCIe atomics in pcie-root-port

PCIe atomics in pcie-root-port

[Robin Voetter]

Hello,

I am trying to use qemu to run ROCm in a virtual machine via GPU 
passthrough with vfio-pci. While this mostly works out of the box, ROCm 
requires PCIe atomics to function properly, and it seems like that this 
is a feature that is missing in qemu. The setup is using a simple PCie 
topology, where the GPU is connected to a pcie-root-port as follows:
    -device pcie-root-port,id=pcie.1
    -device vfio-pci,host=<host_pci_address>,bus=pcie.1

When the amdgpu kernel module is loaded in the guest, enabling PCIe 
atomics fails because it requires that PCIe root ports support 32- and 
64-bit atomic completion, see commits 430a2368 and 8e6d0b69 in Linux.

I patched the required flags into the DevCap2 register in qemu and 
everything seems to work fine. I'm interested in getting this change 
into qemu proper, but I have some questions about how to do that:

1. In order for PCIe atomics to be supported, every bridge in the path 
between the root complex and the device in question needs to support the 
routing capability, as well as the device being required to support the 
atomic requester/completer capabilities of course. When a certain device 
from the host is connected to a root port in qemu, that path will be 
abstracted away into the virtual PCIe topology. Should this property 
from the host be reflected into the virtual path in qemu, or is it 
better to gate this behind a property of the pcie-root-port? For 
example, the user could then simply pass -device 
pcie-root-port,id=pcie.1,atomics=on to make the pcie-root-port report 
that it supports these operations. If this should be reflected from the 
host automatically, how should this be tested for? I checked around the 
source for how for example the link speed is set, but it seems like that 
is not reflected from the host system automatically either.

2. Is anything else required to support PCIe atomics, or did I just get 
lucky that my programs work?

Thanks,

Robin Voetter, Stream HPC

Re: PCIe atomics in pcie-root-port

[Alex Williamson]

On Thu, 6 Apr 2023 11:29:34 +0200
Robin Voetter <robin@streamhpc.com> wrote:

> Hello,
> 
> I am trying to use qemu to run ROCm in a virtual machine via GPU 
> passthrough with vfio-pci. While this mostly works out of the box, ROCm 
> requires PCIe atomics to function properly, and it seems like that this 
> is a feature that is missing in qemu. The setup is using a simple PCie 
> topology, where the GPU is connected to a pcie-root-port as follows:
>     -device pcie-root-port,id=pcie.1
>     -device vfio-pci,host=<host_pci_address>,bus=pcie.1
> 
> When the amdgpu kernel module is loaded in the guest, enabling PCIe 
> atomics fails because it requires that PCIe root ports support 32- and 
> 64-bit atomic completion, see commits 430a2368 and 8e6d0b69 in Linux.
> 
> I patched the required flags into the DevCap2 register in qemu and 
> everything seems to work fine. I'm interested in getting this change 
> into qemu proper, but I have some questions about how to do that:
> 
> 1. In order for PCIe atomics to be supported, every bridge in the path 
> between the root complex and the device in question needs to support the 
> routing capability, as well as the device being required to support the 
> atomic requester/completer capabilities of course. When a certain device 
> from the host is connected to a root port in qemu, that path will be 
> abstracted away into the virtual PCIe topology. Should this property 
> from the host be reflected into the virtual path in qemu, or is it 
> better to gate this behind a property of the pcie-root-port? For 
> example, the user could then simply pass -device 
> pcie-root-port,id=pcie.1,atomics=on to make the pcie-root-port report 
> that it supports these operations. If this should be reflected from the 
> host automatically, how should this be tested for? I checked around the 
> source for how for example the link speed is set, but it seems like that 
> is not reflected from the host system automatically either.

I think the typical approach for QEMU would be expose options in the
downstream ports that would then need to be enabled by the user or
management tool, but that's where the complication begins.  At some
point we would want management tools to "do the right thing"
themselves.  Is support for PCIe atomics pervasive enough to default to
enabling support?  How do we handle hotplugged endpoints where the
interconnects do not expose atomics support, or perhaps when they
expose support that doesn't exist?  At some point in the future when we
have migration of devices with atomics, do we need to test all endpoint
to endpoint paths for equivalent atomic support on the migration target?
Are there capabilities on the endpoint that we can virtualize to
disable use of atomics if the host and guest topologies are out of sync?

> 2. Is anything else required to support PCIe atomics, or did I just get 
> lucky that my programs work?

It's been on my todo list for some time to look into supporting atomic
ops, but it suffers from the -ENOTIME syndome.  Thanks,

Alex

Re: PCIe atomics in pcie-root-port

[Robin Voetter]

On 4/6/23 20:40, Alex Williamson wrote:

> I think the typical approach for QEMU would be expose options in the
> downstream ports that would then need to be enabled by the user or
> management tool, but that's where the complication begins.  At some
> point we would want management tools to "do the right thing"
> themselves.  Is support for PCIe atomics pervasive enough to default to
> enabling support?

Apparently this is supported from Haswell or newer on Intel, and 
Ryzen/Threadripper/Epyc on AMD hardware. I don't have any official data 
for this, though.

> How do we handle hotplugged endpoints where the
> interconnects do not expose atomics support, or perhaps when they
> expose support that doesn't exist?  At some point in the future when we
> have migration of devices with atomics, do we need to test all endpoint
> to endpoint paths for equivalent atomic support on the migration target?
> Are there capabilities on the endpoint that we can virtualize to
> disable use of atomics if the host and guest topologies are out of sync?
It seems automatic detection is a lot of added complication for this 
feature. For the time being, I think its best to allow enabling PCIe 
atomics using a device property for the pcie-root-port that is disabled 
by default. If general hardware support is good enough, it can be 
enabled by default. Compatibility with older QEMU versions can then be 
added using a newer hw compat in hw/core/machine.c.

Kind regards,

Robin