From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:32985) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WB93S-0003mA-TR for qemu-devel@nongnu.org; Wed, 05 Feb 2014 15:26:48 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WB93M-0001P6-S8 for qemu-devel@nongnu.org; Wed, 05 Feb 2014 15:26:42 -0500 Received: from mx1.redhat.com ([209.132.183.28]:14016) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WB93M-0001Oh-Ky for qemu-devel@nongnu.org; Wed, 05 Feb 2014 15:26:36 -0500 Message-ID: <1391631994.15608.7.camel@ul30vt.home> From: Alex Williamson Date: Wed, 05 Feb 2014 13:26:34 -0700 In-Reply-To: <20140205185945.GA996@parallels.com> References: <20140205185945.GA996@parallels.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Multi GPU passthrough via VFIO List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Maik Broemme Cc: qemu-devel@nongnu.org On Wed, 2014-02-05 at 19:59 +0100, Maik Broemme wrote: > Hi, > > currently VFIO with multi GPU passthrough is working partially and > hopefully somebody has a hint about the problem. I'm doing passthrough > of an AMD Radeon R9 290X and AMD Radeon 7870 GHz Edition to a single VM. > > If the VM is running Linux this works quite well with radeon or fglrx > driver. Please see 'dmesg' log attached, when using the radeon driver. > If needed I can also post one with fglrx driver. > > If I do the exact same passthrough to a Windows VM and use latest AMD > Catalyst 14.1 (2/1/2014) or AMD Catalyst 13.12 (12/18/2013) I can get > only the first device working (AMD R9 290X) with 'x-vga=on'. I don't > enable 'x-vga=on' on second device as this should never work. :) Why not? The guest is able to change the VGA enable bit in the emulated bridge registers and access VGA space of each device, just like happens on bare metal. You'll only get one device initialized from seabios, but that's the same as would happen on bare metal as well. > I see > BIOS boot screen and everything works fine except for the second GPU. > The windows device manager just show me "Code 12" for the second GPU > and its HD Audio device. Code 12 means: "This device cannot find enough > free resources that it can use". I've seen the same using Nvidia GRID GPUs (w/o x-vga=on), but only with the Q35 chipset model, Linux works, Windows reports Code 12. I have no idea why as all the PCI resources appear to be properly sized and mapped. FWIW, 2 GRID GPUs assigned to a guest do work with the 440FX chipset model. Beyond 2 we run out of MMIO resources below 4G and something bad happens. > QEMU is called in both cases via the following. I just replace the > '-drive' accordingly. > > /usr/bin/taskset -c 0,1,2,3 /usr/bin/qemu-system-x86_64 \ > -machine q35,accel=kvm \ > -enable-kvm \ > -nodefaults \ > -nographic \ > -vga none \ > -boot order=nc \ > -cpu host \ > -smp cores=4,threads=1,sockets=1 \ > -m 8192 \ > -rtc base=localtime \ > -k de \ > -drive file=/srv/kvm/linux-drive0.img,id=drive0,if=none,cache=none,aio=threads \ > -mon chardev=monitor0 \ > -chardev socket,id=monitor0,path=/tmp/linux.monitor,nowait,server \ > -netdev tap,id=net0,vhost=on,helper=/usr/lib/qemu/qemu-bridge-helper \ > -device virtio-net-pci,netdev=net0,mac=00:00:00:02:01:04 \ > -device virtio-blk-pci,drive=drive0,ioeventfd=on \ > -device ioh3420,bus=pcie.0,id=pcie0,port=1,chassis=1,multifunction=on \ > -device ioh3420,bus=pcie.0,id=pcie1,port=2,chassis=2,multifunction=on \ > -device vfio-pci,host=01:00.0,addr=00.0,bus=pcie0,multifunction=on,x-vga=on \ > -device vfio-pci,host=01:00.1,addr=00.1,bus=pcie0 \ > -device vfio-pci,host=02:00.0,addr=00.0,bus=pcie1,multifunction=on \ > -device vfio-pci,host=02:00.1,addr=00.1,bus=pcie1 \ > -no-reboot > > My setup is the following: > > Kernel: linux-3.13.1 > Seabios: seabios-git-rel.1.7.4.r51.g151d034 (5/2/2014) > QEMU: qemu-git-2.0.r30666.g31db5b3 (5/2/2014) > > Below is the 'lspci' output and I'm using the AMD Radeon HD 5430 as device > for my local X server: > > 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02) > 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) > 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B) > 00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port D) > 00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port H) > 00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port B) > 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40) > 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller > 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller > 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller > 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller > 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42) > 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) > 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) > 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40) > 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller > 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) > 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1) > 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2) > 00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 3) > 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller > 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller > 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 > 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 > 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 > 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 > 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 > 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 > 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon HD 8970] > 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8 > 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn XT [Radeon HD 7870 GHz Edition] > 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] > 03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) > 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] > 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] > 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) > 07:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) > > Another minor issue is that the R9 290X is not reset during shutdown of > VM (neither Linux nor Windows) but it can be tricked with doing > "suspend-to-ram" between two starts. That's why I use '-no-reboot' option > in QEMU. The 7870 is doing the reset properly. Is the NoSoftRst "-" on the 290X vs "+" on the 7870 in lspci -vvv by chance? Thanks, Alex