From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steven DuChene Subject: trying to use vfio to pass VGA card and getting operation not permitted error Date: Thu, 13 Nov 2014 11:50:10 -0500 Message-ID: <5464E142.6010903@hp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit To: kvm@vger.kernel.org Return-path: Received: from g9t1613g.houston.hp.com ([15.240.0.71]:41383 "EHLO g9t1613g.houston.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932245AbaKMQuM (ORCPT ); Thu, 13 Nov 2014 11:50:12 -0500 Received: from g6t1525.atlanta.hp.com (g6t1525.atlanta.hp.com [15.193.200.68]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by g9t1613g.houston.hp.com (Postfix) with ESMTPS id 721AE65373 for ; Thu, 13 Nov 2014 16:50:11 +0000 (UTC) Received: from g5t1633.atlanta.hp.com (g5t1633.atlanta.hp.com [16.201.144.132]) by g6t1525.atlanta.hp.com (Postfix) with ESMTP id 5E2B3174 for ; Thu, 13 Nov 2014 16:50:10 +0000 (UTC) Received: from [10.152.32.86] (ospra0.fc.hp.com [16.79.38.117]) by g5t1633.atlanta.hp.com (Postfix) with ESMTP id 0D97C90 for ; Thu, 13 Nov 2014 16:50:09 +0000 (UTC) Sender: kvm-owner@vger.kernel.org List-ID: I am trying to use VFIO and pci-bind to pass a NVidia VGA card and it's companion audio device through to a VM I am trying to start up. I am trying this on two different hardware platforms, a HP zbook 15 and a HP z800 workstation. Both systems are running Ubuntu 14.04. Each time I try to start the VM on the laptop using "virsh start vmname" I get the following errors: error: Failed to start domain vmname error: internal error: early end of file from monitor: possible problem: qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: vfio: error opening /dev/vfio/1: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: Device 'vfio-pci' could not be initialized If I start the VM from the command line on the z800 as root using the qemu-system-x86_64 command directly it works. The VM starts and both of the passed through devices show up in the VM. If I try to start the same VM as the libvirt-qemu user using the same qemu-system-x86_64 command I get the same above error (but with a different group number obviously). My order of operations are: blacklist the nouveau driver from loading (the closed source nvidia driver is not loaded on either system) In the kernel command line passed through grub on the z800 workstation I have: intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 on the zbook15 laptop I have: pci-stub.ids=10de:11fc,10de:0e0b intel_iommu=on and on the laptop I have added the "allow_unsafe_interrupts=1" option to the modprobe of that module. On the z800 I manually bind the nvidia video and audio devices to the pci-stub driver. In both cases I see the following in the kernel dmesg: (from the laptop) [ 6.342603] pci-stub: add 10DE:11FC sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 6.342618] pci-stub 0000:01:00.0: claimed by stub [ 6.342625] pci-stub: add 10DE:0E0B sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 6.342632] pci-stub 0000:01:00.1: claimed by stub (or from the z800 workstation) [ 115.116860] pci-stub 0000:0f:00.1: claimed by stub [ 157.126503] pci-stub 0000:0f:00.0: claimed by stub I then use a vfio-bind script to bind both the video and audio devices to the vfio driver. Once I do that I see the appropriate files under /dev/vfio (on the laptop) drwxr-xr-x 2 root root 80 Nov 13 08:15 ./ drwxr-xr-x 18 root root 4380 Nov 13 08:14 ../ crw------- 1 root root 249, 1 Nov 13 08:15 1 crw-rw-rw- 1 root root 249, 0 Nov 13 08:14 vfio (on the z800) drwxr-xr-x 2 root root 80 Nov 13 10:26 ./ drwxr-xr-x 16 root root 4540 Nov 13 10:26 ../ crw------- 1 root root 247, 1 Nov 13 10:26 14 crw-rw-rw- 1 root root 247, 0 Nov 13 10:26 vfio I have confirmed that the only devices in the iommu group is the nvidia video and audio devices and I am attempting to pass both devices through to the VM when I invoke it. On the laptop I was seeing messages in the logs from apparmor each time I tried to start the VM referring to a libvirt profile for this VM's uuid. After looking into that for a while, I finally set the security_driver in /etc/libvirt/qemu.conf to "none" and that stopped those messages from showing up in the logs. This also prevented the libvirt-UUID files from showing up for this VM in /etc/apparmor.d/libvirt/ each time I tried to start the VM. I did try to add the files in /dev/vfio to the apparmor profile and TEMPLATE files but that did not seem to have any effect on the log messages. In both cases I am actually wanting to start the VMs though openstack which invokes the qemu-system-x86_64 process through libvirt. However if I try to start the VM on the z800 workstation as root from the command line using the following command: qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 2,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=sven.1 -device vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0 -device vfio-pci,host=01:00.1,bus=sven.1,addr=00.1 -drive file=/root/ubuntu_withdriver.raw,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk -drive file=/root/my-seed.img,id=isocd -device ide-cd,bus=ide.1,drive=isocd -vnc 0.0.0.0:1 -k en-us it works. If I try to start the VM on the z800 as the libvirt-qemu user (user id virsh and openstack use) I get the error I listed at the top of this E-mail. I have even tried changing the ownerships of the files in /dev/vfio to this user but that did not make any difference and I still get the same error. I have spent quite a lot of time on this doing lots of research and I seem to be no closer to solving the problem. Can any experts on using VFIO see any problems with the steps and configurations I have done or know of any other reason why I might be getting this error sequence? Any pointers to additional documentation or suggestions would be most appreciated. -- Steven DuChene