From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steven DuChene Subject: Re: trying to use vfio to pass VGA card and getting operation not permitted error Date: Thu, 13 Nov 2014 12:36:42 -0500 Message-ID: <5464EC2A.4090606@hp.com> References: <5464E142.6010903@hp.com> <1415899488.27937.29.camel@ul30vt.home> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org To: Alex Williamson Return-path: Received: from g9t1613g.houston.hp.com ([15.240.0.71]:48017 "EHLO g9t1613g.houston.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751116AbaKMRgr (ORCPT ); Thu, 13 Nov 2014 12:36:47 -0500 Received: from g5t1625.atlanta.hp.com (g5t1625.atlanta.hp.com [15.192.137.8]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by g9t1613g.houston.hp.com (Postfix) with ESMTPS id C3E0B654BB for ; Thu, 13 Nov 2014 17:36:43 +0000 (UTC) In-Reply-To: <1415899488.27937.29.camel@ul30vt.home> Sender: kvm-owner@vger.kernel.org List-ID: Alex: Thanks for the quick reply. Yes, I am using qemu:args in the xml file. I looked around via google for an example of how to use hostdev in the xml file for the audio or video devices. I seem to recall the only example I could find was a hostdev section in the libvirt wiki for a NIC card. Do you know of any example xml files where hostdev is used for video and audio devices? -- Steven DuChene On 11/13/2014 12:24 PM, Alex Williamson wrote: > On Thu, 2014-11-13 at 11:50 -0500, Steven DuChene wrote: >> I am trying to use VFIO and pci-bind to pass a NVidia VGA card and it's >> companion audio device through to a VM I am trying to start up. >> >> I am trying this on two different hardware platforms, a HP zbook 15 and >> a HP z800 workstation. Both systems are running Ubuntu 14.04. >> >> Each time I try to start the VM on the laptop using "virsh start vmname" >> I get the following errors: >> >> error: Failed to start domain vmname >> error: internal error: early end of file from monitor: possible problem: >> qemu-system-x86_64: -device >> vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: >> vfio: error opening /dev/vfio/1: Operation not permitted >> qemu-system-x86_64: -device >> vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: >> vfio: failed to get group 1 >> qemu-system-x86_64: -device >> vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: >> Device initialization failed. >> qemu-system-x86_64: -device >> vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: >> Device 'vfio-pci' could not be initialized >> >> If I start the VM from the command line on the z800 as root using the >> qemu-system-x86_64 command directly it works. The VM starts and both of >> the passed through devices show up in the VM. If I try to start the same >> VM as the libvirt-qemu user using the same qemu-system-x86_64 command I >> get the same above error (but with a different group number obviously). >> >> My order of operations are: >> >> blacklist the nouveau driver from loading (the closed source nvidia >> driver is not loaded on either system) >> >> In the kernel command line passed through grub on the z800 workstation I >> have: >> >> intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 >> >> on the zbook15 laptop I have: >> >> pci-stub.ids=10de:11fc,10de:0e0b intel_iommu=on >> >> and on the laptop I have added the "allow_unsafe_interrupts=1" option to >> the modprobe of that module. >> >> On the z800 I manually bind the nvidia video and audio devices to the >> pci-stub driver. >> >> In both cases I see the following in the kernel dmesg: >> >> (from the laptop) >> [ 6.342603] pci-stub: add 10DE:11FC sub=FFFFFFFF:FFFFFFFF >> cls=00000000/00000000 >> [ 6.342618] pci-stub 0000:01:00.0: claimed by stub >> [ 6.342625] pci-stub: add 10DE:0E0B sub=FFFFFFFF:FFFFFFFF >> cls=00000000/00000000 >> [ 6.342632] pci-stub 0000:01:00.1: claimed by stub >> >> (or from the z800 workstation) >> [ 115.116860] pci-stub 0000:0f:00.1: claimed by stub >> [ 157.126503] pci-stub 0000:0f:00.0: claimed by stub >> >> I then use a vfio-bind script to bind both the video and audio devices >> to the vfio driver. >> Once I do that I see the appropriate files under /dev/vfio >> >> (on the laptop) >> drwxr-xr-x 2 root root 80 Nov 13 08:15 ./ >> drwxr-xr-x 18 root root 4380 Nov 13 08:14 ../ >> crw------- 1 root root 249, 1 Nov 13 08:15 1 >> crw-rw-rw- 1 root root 249, 0 Nov 13 08:14 vfio >> >> (on the z800) >> drwxr-xr-x 2 root root 80 Nov 13 10:26 ./ >> drwxr-xr-x 16 root root 4540 Nov 13 10:26 ../ >> crw------- 1 root root 247, 1 Nov 13 10:26 14 >> crw-rw-rw- 1 root root 247, 0 Nov 13 10:26 vfio >> >> I have confirmed that the only devices in the iommu group is the nvidia >> video and audio devices and I am attempting to pass both devices through >> to the VM when I invoke it. >> >> On the laptop I was seeing messages in the logs from apparmor each time >> I tried to start the VM referring to a libvirt profile for this VM's >> uuid. After looking into that for a while, I finally set the >> security_driver in /etc/libvirt/qemu.conf to "none" and that stopped >> those messages from showing up in the logs. This also prevented the >> libvirt-UUID files from showing up for this VM in >> /etc/apparmor.d/libvirt/ each time I tried to start the VM. I did try to >> add the files in /dev/vfio to the apparmor profile and TEMPLATE files >> but that did not seem to have any effect on the log messages. >> >> In both cases I am actually wanting to start the VMs though openstack >> which invokes the qemu-system-x86_64 process through libvirt. However if >> I try to start the VM on the z800 workstation as root from the command >> line using the following command: >> >> qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp >> 2,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -device >> ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=sven.1 >> -device >> vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0 >> -device vfio-pci,host=01:00.1,bus=sven.1,addr=00.1 -drive >> file=/root/ubuntu_withdriver.raw,id=disk,format=raw -device >> ide-hd,bus=ide.0,drive=disk -drive file=/root/my-seed.img,id=isocd >> -device ide-cd,bus=ide.1,drive=isocd -vnc 0.0.0.0:1 -k en-us >> >> it works. If I try to start the VM on the z800 as the libvirt-qemu user >> (user id virsh and openstack use) I get the error I listed at the top of >> this E-mail. I have even tried changing the ownerships of the files in >> /dev/vfio to this user but that did not make any difference and I still >> get the same error. >> >> I have spent quite a lot of time on this doing lots of research and I >> seem to be no closer to solving the problem. >> >> Can any experts on using VFIO see any problems with the steps and >> configurations I have done or know of any other reason why I might be >> getting this error sequence? > What does your domain xml look for like assigning the device? I suspect > you might be using a block to define the assigned device so > that you can add the x-vga=on option, which unfortunately has the > side-effect that libvirt is no longer aware of the device and cannot set > file permissions and locked memory limits correctly for doing device > assignment. If so, you might be able to assign the audio function using > a standard section, which should setup the permissions for the > GPU in a section as well since they're in the same group. > Another option is to use a wrapper script around qemu, for instance: > > /usr/libexec/qemu-kvm-nvidia-vga > > # cat /usr/libexec/qemu-kvm-nvidia-vga > #!/bin/sh > exec /usr/libexec/qemu-kvm \ > `echo "\$@" | sed 's|01:00.0|01:00.0,x-vga=on|g'` > > You'll likely also need to update apparmor to allow libvirt to execute > this script. For selinux you'd do something like described in this > document: > > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Technical_Notes/virt.html > > The wrapper script option would allow both devices in the group to be > fully managed by libvirt and would then remove your need to manually run > any sort of vfio-bind script. > > However, I've now looked up the quickspecs for the zbook and see that > you're likely using an Nvidia Quadro K-series GPU and hopefully > something similar on the z800. These can be used as secondary GPUs in > the VM and do not need the x-vga option. Simply attach the GPU and > audio to the VM (using , not ) in addition to one of > the emulated VGA models. The emulated VGA will be used for boot and > install and the assigned GPU will be activated when the Nvidia driver > stack is loaded. > > Caveats: a) I have no idea if Ubuntu 14.04 includes the necessary > support for GPU assignment. b) Laptop do strange and bizarre things > with discrete GPUs supporting Optimus. It may not work, and if it does > work, it may not output through any of the external video ports on the > laptop or the screen. You might want to setup remote access to the VM > to access the desktop via network and not rely on a physical output > port. > > Good luck, > Alex >