Re: trying to use vfio to pass VGA card and getting operation not permitted error

[Steven DuChene <steven.duchene@hp.com>]

Alex:
Thanks for the quick reply.
Yes, I am using qemu:args in the xml file.
I looked around via google for an example of how to use hostdev in the 
xml file for the audio or video devices.
I seem to recall the only example I could find was a hostdev section in 
the libvirt wiki for a NIC card.
Do you know of any example xml files where hostdev is used for video and 
audio devices?
--
Steven DuChene

On 11/13/2014 12:24 PM, Alex Williamson wrote:
> On Thu, 2014-11-13 at 11:50 -0500, Steven DuChene wrote:
>> I am trying to use VFIO and pci-bind to pass a NVidia VGA card and it's
>> companion audio device through to a VM I am trying to start up.
>>
>> I am trying this on two different hardware platforms, a HP zbook 15 and
>> a HP z800 workstation. Both systems are running Ubuntu 14.04.
>>
>> Each time I try to start the VM on the laptop using "virsh start vmname"
>> I get the following errors:
>>
>> error: Failed to start domain vmname
>> error: internal error: early end of file from monitor: possible problem:
>> qemu-system-x86_64: -device
>> vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0:
>> vfio: error opening /dev/vfio/1: Operation not permitted
>> qemu-system-x86_64: -device
>> vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0:
>> vfio: failed to get group 1
>> qemu-system-x86_64: -device
>> vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0:
>> Device initialization failed.
>> qemu-system-x86_64: -device
>> vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0:
>> Device 'vfio-pci' could not be initialized
>>
>> If I start the VM from the command line on the z800 as root using the
>> qemu-system-x86_64 command directly it works. The VM starts and both of
>> the passed through devices show up in the VM. If I try to start the same
>> VM as the libvirt-qemu user using the same qemu-system-x86_64 command I
>> get the same above error (but with a different group number obviously).
>>
>> My order of operations are:
>>
>> blacklist the nouveau driver from loading (the closed source nvidia
>> driver is not loaded on either system)
>>
>> In the kernel command line passed through grub on the z800 workstation I
>> have:
>>
>> intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
>>
>> on the zbook15 laptop I have:
>>
>> pci-stub.ids=10de:11fc,10de:0e0b intel_iommu=on
>>
>> and on the laptop I have added the "allow_unsafe_interrupts=1" option to
>> the modprobe of that module.
>>
>> On the z800 I manually bind the nvidia video and audio devices to the
>> pci-stub driver.
>>
>> In both cases I see the following in the kernel dmesg:
>>
>> (from the laptop)
>>    [    6.342603] pci-stub: add 10DE:11FC sub=FFFFFFFF:FFFFFFFF
>> cls=00000000/00000000
>> [    6.342618] pci-stub 0000:01:00.0: claimed by stub
>> [    6.342625] pci-stub: add 10DE:0E0B sub=FFFFFFFF:FFFFFFFF
>> cls=00000000/00000000
>> [    6.342632] pci-stub 0000:01:00.1: claimed by stub
>>
>> (or from the z800 workstation)
>> [  115.116860] pci-stub 0000:0f:00.1: claimed by stub
>> [  157.126503] pci-stub 0000:0f:00.0: claimed by stub
>>
>> I then use a vfio-bind script to bind both the video and audio devices
>> to the vfio driver.
>> Once I do that I see the appropriate files under /dev/vfio
>>
>> (on the laptop)
>> drwxr-xr-x  2 root root     80 Nov 13 08:15 ./
>> drwxr-xr-x 18 root root   4380 Nov 13 08:14 ../
>> crw-------  1 root root 249, 1 Nov 13 08:15 1
>> crw-rw-rw-  1 root root 249, 0 Nov 13 08:14 vfio
>>
>> (on the z800)
>> drwxr-xr-x  2 root root     80 Nov 13 10:26 ./
>> drwxr-xr-x 16 root root   4540 Nov 13 10:26 ../
>> crw-------  1 root root 247, 1 Nov 13 10:26 14
>> crw-rw-rw-  1 root root 247, 0 Nov 13 10:26 vfio
>>
>> I have confirmed that the only devices in the iommu group is the nvidia
>> video and audio devices and I am attempting to pass both devices through
>> to the VM when I invoke it.
>>
>> On the laptop I was seeing messages in the logs from apparmor each time
>> I tried to start the VM referring to a libvirt profile for this VM's
>> uuid. After looking into that for a while, I finally set the
>> security_driver in /etc/libvirt/qemu.conf to "none" and that stopped
>> those messages from showing up in the logs. This also prevented the
>> libvirt-UUID files from showing up for this VM in
>> /etc/apparmor.d/libvirt/ each time I tried to start the VM. I did try to
>> add the files in /dev/vfio to the apparmor profile and TEMPLATE files
>> but that did not seem to have any effect on the log messages.
>>
>> In both cases I am actually wanting to start the VMs though openstack
>> which invokes the qemu-system-x86_64 process through libvirt. However if
>> I try to start the VM on the z800 workstation as root from the command
>> line using the following command:
>>
>> qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp
>> 2,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -device
>> ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=sven.1
>> -device
>> vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0
>> -device vfio-pci,host=01:00.1,bus=sven.1,addr=00.1 -drive
>> file=/root/ubuntu_withdriver.raw,id=disk,format=raw -device
>> ide-hd,bus=ide.0,drive=disk -drive file=/root/my-seed.img,id=isocd
>> -device ide-cd,bus=ide.1,drive=isocd -vnc 0.0.0.0:1 -k en-us
>>
>> it works. If I try to start the VM on the z800 as the libvirt-qemu user
>> (user id virsh and openstack use) I get the error I listed at the top of
>> this E-mail. I have even tried changing the ownerships of the files in
>> /dev/vfio to this user but that did not make any difference and I still
>> get the same error.
>>
>> I have spent quite a lot of time on this doing lots of research and I
>> seem to be no closer to solving the problem.
>>
>> Can any experts on using VFIO see any problems with the steps and
>> configurations I have done or know of any other reason why I might be
>> getting this error sequence?
> What does your domain xml look for like assigning the device?  I suspect
> you might be using a <qemu:args> block to define the assigned device so
> that you can add the x-vga=on option, which unfortunately has the
> side-effect that libvirt is no longer aware of the device and cannot set
> file permissions and locked memory limits correctly for doing device
> assignment.  If so, you might be able to assign the audio function using
> a standard <hostdev> section, which should setup the permissions for the
> GPU in a <qemu:args> section as well since they're in the same group.
> Another option is to use a wrapper script around qemu, for instance:
>
>   <emulator>/usr/libexec/qemu-kvm-nvidia-vga</emulator>
>
> # cat /usr/libexec/qemu-kvm-nvidia-vga
> #!/bin/sh
> exec /usr/libexec/qemu-kvm \
> 	`echo "\$@" | sed 's|01:00.0|01:00.0,x-vga=on|g'`
>
> You'll likely also need to update apparmor to allow libvirt to execute
> this script.  For selinux you'd do something like described in this
> document:
>
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Technical_Notes/virt.html
>
> The wrapper script option would allow both devices in the group to be
> fully managed by libvirt and would then remove your need to manually run
> any sort of vfio-bind script.
>
> However, I've now looked up the quickspecs for the zbook and see that
> you're likely using an Nvidia Quadro K-series GPU and hopefully
> something similar on the z800.  These can be used as secondary GPUs in
> the VM and do not need the x-vga option.  Simply attach the GPU and
> audio to the VM (using <hostdev>, not <qemu:arg>) in addition to one of
> the emulated VGA models.  The emulated VGA will be used for boot and
> install and the assigned GPU will be activated when the Nvidia driver
> stack is loaded.
>
> Caveats: a) I have no idea if Ubuntu 14.04 includes the necessary
> support for GPU assignment.  b) Laptop do strange and bizarre things
> with discrete GPUs supporting Optimus.  It may not work, and if it does
> work, it may not output through any of the external video ports on the
> laptop or the screen.  You might want to setup remote access to the VM
> to access the desktop via network and not rely on a physical output
> port.
>
> Good luck,
> Alex
>