trying to use vfio to pass VGA card and getting operation not permitted error

[Steven DuChene <steven.duchene@hp.com>]

I am trying to use VFIO and pci-bind to pass a NVidia VGA card and it's 
companion audio device through to a VM I am trying to start up.

I am trying this on two different hardware platforms, a HP zbook 15 and 
a HP z800 workstation. Both systems are running Ubuntu 14.04.

Each time I try to start the VM on the laptop using "virsh start vmname" 
I get the following errors:

error: Failed to start domain vmname
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device 
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: 
vfio: error opening /dev/vfio/1: Operation not permitted
qemu-system-x86_64: -device 
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: 
vfio: failed to get group 1
qemu-system-x86_64: -device 
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: 
Device initialization failed.
qemu-system-x86_64: -device 
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: 
Device 'vfio-pci' could not be initialized

If I start the VM from the command line on the z800 as root using the 
qemu-system-x86_64 command directly it works. The VM starts and both of 
the passed through devices show up in the VM. If I try to start the same 
VM as the libvirt-qemu user using the same qemu-system-x86_64 command I 
get the same above error (but with a different group number obviously).

My order of operations are:

blacklist the nouveau driver from loading (the closed source nvidia 
driver is not loaded on either system)

In the kernel command line passed through grub on the z800 workstation I 
have:

intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1

on the zbook15 laptop I have:

pci-stub.ids=10de:11fc,10de:0e0b intel_iommu=on

and on the laptop I have added the "allow_unsafe_interrupts=1" option to 
the modprobe of that module.

On the z800 I manually bind the nvidia video and audio devices to the 
pci-stub driver.

In both cases I see the following in the kernel dmesg:

(from the laptop)
  [    6.342603] pci-stub: add 10DE:11FC sub=FFFFFFFF:FFFFFFFF 
cls=00000000/00000000
[    6.342618] pci-stub 0000:01:00.0: claimed by stub
[    6.342625] pci-stub: add 10DE:0E0B sub=FFFFFFFF:FFFFFFFF 
cls=00000000/00000000
[    6.342632] pci-stub 0000:01:00.1: claimed by stub

(or from the z800 workstation)
[  115.116860] pci-stub 0000:0f:00.1: claimed by stub
[  157.126503] pci-stub 0000:0f:00.0: claimed by stub

I then use a vfio-bind script to bind both the video and audio devices 
to the vfio driver.
Once I do that I see the appropriate files under /dev/vfio

(on the laptop)
drwxr-xr-x  2 root root     80 Nov 13 08:15 ./
drwxr-xr-x 18 root root   4380 Nov 13 08:14 ../
crw-------  1 root root 249, 1 Nov 13 08:15 1
crw-rw-rw-  1 root root 249, 0 Nov 13 08:14 vfio

(on the z800)
drwxr-xr-x  2 root root     80 Nov 13 10:26 ./
drwxr-xr-x 16 root root   4540 Nov 13 10:26 ../
crw-------  1 root root 247, 1 Nov 13 10:26 14
crw-rw-rw-  1 root root 247, 0 Nov 13 10:26 vfio

I have confirmed that the only devices in the iommu group is the nvidia 
video and audio devices and I am attempting to pass both devices through 
to the VM when I invoke it.

On the laptop I was seeing messages in the logs from apparmor each time 
I tried to start the VM referring to a libvirt profile for this VM's 
uuid. After looking into that for a while, I finally set the 
security_driver in /etc/libvirt/qemu.conf to "none" and that stopped 
those messages from showing up in the logs. This also prevented the 
libvirt-UUID files from showing up for this VM in 
/etc/apparmor.d/libvirt/ each time I tried to start the VM. I did try to 
add the files in /dev/vfio to the apparmor profile and TEMPLATE files 
but that did not seem to have any effect on the log messages.

In both cases I am actually wanting to start the VMs though openstack 
which invokes the qemu-system-x86_64 process through libvirt. However if 
I try to start the VM on the z800 workstation as root from the command 
line using the following command:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 
2,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -device 
ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=sven.1 
-device 
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0 
-device vfio-pci,host=01:00.1,bus=sven.1,addr=00.1 -drive 
file=/root/ubuntu_withdriver.raw,id=disk,format=raw -device 
ide-hd,bus=ide.0,drive=disk -drive file=/root/my-seed.img,id=isocd 
-device ide-cd,bus=ide.1,drive=isocd -vnc 0.0.0.0:1 -k en-us

it works. If I try to start the VM on the z800 as the libvirt-qemu user 
(user id virsh and openstack use) I get the error I listed at the top of 
this E-mail. I have even tried changing the ownerships of the files in 
/dev/vfio to this user but that did not make any difference and I still 
get the same error.

I have spent quite a lot of time on this doing lots of research and I 
seem to be no closer to solving the problem.

Can any experts on using VFIO see any problems with the steps and 
configurations I have done or know of any other reason why I might be 
getting this error sequence?

Any pointers to additional documentation or suggestions would be most 
appreciated.
--
Steven DuChene