Re: [Qemu-devel] AMD video card passthrough reset issues

["Lucio Andrés Illanes Albornoz" <l.illanes@gmx.de>]

On Tue, 02 Dec 2014 08:26:20 -0700 Alex Williamson <alex.williamson@redhat.com> wrote:
> All of the Bonaire-based AMD GPUs seems to have issues with reset
> (R7790, R7 260/X).  I've tried to engage AMD on this, but haven't gotten
> any response on this topic yet.  For devices like this that don't
> support any kind of function level reset (FLR), VFIO will try to do a
> PCI bus reset on guest reboot.  This is as close as we can get to how
> the BIOS resets the device on a host reboot.  Unfortunately on these
> cards there seems to be some sort of disconnect between the PCI bus
> interface reset and resetting the rest of the GPU.  I believe I've even
> seen cases where a PCI bus reset appears to have no affect on the GPU
> when running in VGA mode.  My best guess is that some firmware running
> in the card isn't clearing itself on reset an attempting to reload it
> causes errors.  Note that a guest can be reset multiple times and the
> device continues to work if the guest is restricted to standard VGA
> drivers (in VGA passthrough mode of course).
My experience is consistent with that description; the bus reset initiated through the hotplug reset interface appears to leave whichever part(s) of my video card in a state the AMD driver is not prepared to handle upon 2nd bootup (e.g. first VM reboot) and thereafter, it's completely gone: endless amounts of IOTLB_INV_TIMEOUT and `Completion-Wait loop timed out' kernel messages and particularly, no VGA output at all when doing primary passthrough (which I no longer require since vgacon isn't too fond of that,) and possibly even hangs upon running lspci (8) afterwards (if I remember correctly, that is.) 

I had originally intended to have QEMU trace MMIO in general and PCI{,-E} bus/device traffic (as relevant) in order to establish what arcane incantations Windows could possibly be performing, but that only ended up showing me PCI configuration space read I/O and IRQ reassignments upon disabling my video card; WinDbg/Kd* is far too slow to facilitate tracing PCI{,-E} traffic through breakpoints and were I to possess the Windows Research Kernel source code, speaking completely hypothetically here, I would then unfortunately have to find out that QEMU w/ KVM plus AMD's drivers doesn't go along too well w/ Windows Server 2003. I then figured that having drivers/vfio/pci/* produce that information should ultimately lead me towards the solution but I can't quite see to that just yet; the remove/rescan dance is the only thing that, pragmatically speaking, actually works for me at present.

> In your experiment with removing and rescanning the device, are you
> simply doing 'echo 1 > remove; echo 1 > /sys/bus/pci/rescan'?  Thanks,
Yes.