Re: [Qemu-devel] QEMU PCIe link "negotiation"

[Alex Williamson <alex.williamson@redhat.com>]

On Tue, 16 Oct 2018 10:49:25 +0100
"Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:

> * Alex Williamson (alex.williamson@redhat.com) wrote:
> > This email is already too long, but I also wonder whether we should
> > consider additional vfio-pci interfaces to trigger a link retraining or
> > allow virtualized access to the physical upstream port config space.
> > Clearly we need to consider multi-function devices and whether there
> > are useful configurations that could benefit from such access.  Thanks
> > for reading, please discuss,  
> 
> I'm not sure about the consequences of the actual link speeds, but I
> worry we'll hit things looking for PCIe v3 features; in particular
> AMD's ROCm code needs PCIe atomics:
> 
> https://github.com/RadeonOpenCompute/ROCm_Documentation/blob/master/Installation_Guide/More-about-how-ROCm-uses-PCIe-Atomics.rst
> 
> so it feels like getting that to work with passthrough would
> need some negotiation of features.

Taking a quick read through the AtomicOps ECN, I think this is
somewhat orthogonal to the link speeds and widths.  Support for acting
as a completer or router of atomic ops is indicated through the DEVCAP2
register in the PCIe capability, it's optional and should not be
assumed by other newer PCIe features.  So I think we can tackle it
separately, but indeed it does appear to be a difficult feature to
implement correctly for a VM, at least if we attempt to do it
automatically.  We might need to burden the user with this sort of
configuration unless AtomicOps support is so ubiquitous that we can
correctly assume that it's available between arbitrary endpoints which
might be assigned to the same VM.  Probably a device option on the
pcie-root-port device to expose atomic op support is possible in the
short term, though more thorough reading of the spec is required.
Thanks,

Alex