Host hard lockup when rebooting VM with GPU assigned via VT-d

[Stijn Tintel <stijn@linux-ipv6.be>]

While I know that assignment of graphics cards isn't officially
supported, I'm reporting this problem nonetheless. This was suggested by
danpb on irc.oftc.net/virt.

I am able to pass through an ATI HD7970 via VT-d, to a VM, and the guest
OS can successfully use it. I tested this with a Windows 7 VM, and with
an Ubuntu 12.10 and a Gentoo VM. I was able to run games or use OpenCL
in the VMs. Unfortunately, when I reboot the VM, the host system
completely locks up. Even the reset button no longer responds as usual
(when I hit reset, the system does nothing for ~15s, then powers down,
and powers up again a few seconds later).

There are a few exceptions:
- when I shut down the VM, suspend and resume the host, then boot the VM
again, the host doesn't lock
- if I don't start X in the Linux VM, I can reboot it as much as I want,
the host doesn't lock
- with the Ubuntu 12.10 VM, and older fglrx driver, I could reboot the
VM without the host locking up, but am unable to use the GPU in the VM
until I reboot or suspend/resume the host

I suspect this problem happens because the GPU BIOS is left in a bad
state after shutting down the VM, and the ATI driver not being able to
handle that. To confirm this I tried to save the state of the GPU in the
VM with vbetool before starting X, but I did not succeed, so I couldn't
try restoring it either. And this wouldn't work for Windows anyway.

This happens with any host kernel between 3.0 and 3.8 I have tested, and
with qemu-kvm 1.0.1 and 1.1.1, qemu 1.1.1, 1.1.2, 1.2.1, 1.3.0 and 1.4.0.

Hardware information:
ASUS P9X79 WS
Intel Core i7-3930K
Gainward GTX480 used by the host OS
ASUS HD7970 passed through with VT-D

If anyone thinks of something I can try to work around (or solve) this
problem, I would very much appreciate hearing about it. Should you need
more detailed information, don't hesitate to ask.

Lastly I want to thank everybody involved with KVM for delivering such
an awesome, open and free product.

Kind regards,
Stijn