From mboxrd@z Thu Jan 1 00:00:00 1970 From: Don Dutile Subject: Re: [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy Date: Mon, 19 Dec 2011 15:32:18 -0500 Message-ID: <4EEF9F52.4030405@redhat.com> References: <201112121115.pBCBFMTA004221@mail.maya.org> <4EE649B4.2000008@redhat.com> <20111213094029.2155fa4b@dualc.maya.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org To: Andreas Hartmann Return-path: Received: from mx1.redhat.com ([209.132.183.28]:31469 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752619Ab1LSUcW (ORCPT ); Mon, 19 Dec 2011 15:32:22 -0500 In-Reply-To: <20111213094029.2155fa4b@dualc.maya.org> Sender: kvm-owner@vger.kernel.org List-ID: On 12/13/2011 03:40 AM, Andreas Hartmann wrote: > Hello Don! > Andreas, sorry for the delay, I don't follow this email list frequently (read: !daily). > Am Tue, 13 Dec 2011 01:21:41 +0100 > schrieb Andreas Hartmann: > >> Am Mon, 12 Dec 2011 13:36:36 -0500 >> schrieb Don Dutile: >> >>> On 12/12/2011 06:15 AM, Andreas Hartmann wrote: >>>> Hello! >>>> >>>> I've got a few questions to a problem, which already was analyzed here >>>> sometime ago: >>>> http://markmail.org/message/dspovwvzp3wtdrf6#query:+page:1+mid:i2oph4xwfmiknt3y+state:results >>>> >>>> My situation is a bit different. I do have two PCI cards (a Linksys wlan >>>> card and an intel e100 card). Each of these cards should be managed in >>>> an own VM. I do have no problems with IRQ sharing (each device does have its own IRQ). >>>> >>>> I'm using linux 3.0.6, kvm 0.15 and libvirt 0.9.7. Mainboard is: GA-990XA-UD3. >>>> >>>> >>>> The problem is: both cards are behind a PCI-PCI bridge: >>>> >>>> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40) >>>> >>>> >>>> -[0000:00]-+-00.0 ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B) >>>> +-00.2 ATI Technologies Inc Device 5a23 >>>> +-02.0-[01]--+-00.0 ATI Technologies Inc Device 6759 >>>> | \-00.1 ATI Technologies Inc Device aa90 >>>> +-04.0-[02]----00.0 Device 1b6f:7023 >>>> +-09.0-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller >>>> +-0a.0-[04]----00.0 Device 1b6f:7023 >>>> +-11.0 ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] >>>> +-12.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller >>>> +-12.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller >>>> +-13.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller >>>> +-13.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller >>>> +-14.0 ATI Technologies Inc SBx00 SMBus Controller >>>> +-14.1 ATI Technologies Inc SB700/SB800 IDE Controller >>>> +-14.2 ATI Technologies Inc SBx00 Azalia (Intel HDA) >>>> +-14.3 ATI Technologies Inc SB700/SB800 LPC host controller >>>> >>>> +-14.4-[05]--+-06.0 Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 >>>> | \-07.0 RaLink RT2800 802.11n PCI >>>> >>> You cannot assign two devices behind a (legacy) PCI (not PCIe) bridge to two >>> different VMs because PCI devices dont provide a B:D.F in their transaction headers; >>> In this case, the PCIe-PCI bridge/switch prepends the bridge's B:D.F in front of >>> a transaction that is sourced by 05:06.0 or 05:07.0 . >>> >>> So, the devices cannot be isolated from each other's (mem) mapping domains in the >>> IOMMU, thus, libvirt prevents this (security) violation from being enabled. >> >> Ok. If I remove the intel card and put in instead a 32 bit PCIe card >> like TP-Link TG-3468, I could assign each of these two cards to >> different VMs. >> KISS: Don't try to assign legacy PCI devices; just PCIe. ;-) Where is "TP-Link TG-3468" in lspci output below? the AMD devices in 18.0->18.5 ??? >> Is this correct? >> >> [...] >> >>>> 1. Is there any way to get the devices into different VMs? (I can't put them to another PCI slot as there are just 2 PCI slots on the board.) >>> Only if the two slots are behind different PCIe-PCI bridges . >>> >>>> 2. Is there any fix to prevent the host crash - maybe in a newer version of kvm or kernel? >>> don't assign the PCI bridge; libvirt does all the proper bind/unbinding of devices >>> for assigned devices, so the above echo steps are unnecessary if you use libvirt. >> >> If I just remove the intel card, I should be able to assign the wlan >> card to one of my VMs. I'll try that. > > Meanwhile I checked this scenario: I removed the intel card and > rebooted. I got the following pci tree: > > -[0000:00]-+-00.0 ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B) > +-00.2 ATI Technologies Inc Device 5a23 > +-02.0-[01]--+-00.0 ATI Technologies Inc Device 6759 > | \-00.1 ATI Technologies Inc Device aa90 > +-04.0-[02]----00.0 Device 1b6f:7023 > +-09.0-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller > +-0a.0-[04]----00.0 Device 1b6f:7023 > +-11.0 ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] > +-12.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller > +-12.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller > +-13.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller > +-13.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller > +-14.0 ATI Technologies Inc SBx00 SMBus Controller > +-14.1 ATI Technologies Inc SB700/SB800 IDE Controller > +-14.2 ATI Technologies Inc SBx00 Azalia (Intel HDA) > +-14.3 ATI Technologies Inc SB700/SB800 LPC host controller > +-14.4-[05]----07.0 RaLink RT2800 802.11n PCI > > +-14.5 ATI Technologies Inc SB700/SB800 USB OHCI2 Controller > +-15.0-[06]-- > +-16.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller > +-16.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller > +-18.0 Advanced Micro Devices [AMD] Device 1600 > +-18.1 Advanced Micro Devices [AMD] Device 1601 > +-18.2 Advanced Micro Devices [AMD] Device 1602 > +-18.3 Advanced Micro Devices [AMD] Device 1603 > +-18.4 Advanced Micro Devices [AMD] Device 1604 > \-18.5 Advanced Micro Devices [AMD] Device 1605 > > But this doesn't work, too: I had to manually unbind the bridge (as described in the mail before - it was > not necessary to manually unbind the device itself). On closing the VM, > the complete host crashed with this screen (excerpt): > You cannot assign a PCI bridge. In fact, you don't need to for device-assignment. You just need to assign the device itself; nothing else in the host PCI hierarchy. .... assuming the device is a PCIe device, and not a graphics card/device. again, libvirt does all the bind/unbinding you need to do. You want to do a 'virsh device-attach pci.xml with pci.xml looking something like:
where XX is replaced by the value of the slot your PCIe device is plugged into. > amd_iommu_domain_destroy > iommu_domain_free > kvm_iommu_unmap_guest > kvm_arch_destroy_vm > kvm_destroy_vm > kvm_vcpu_release > __fput > flip_close > put_files_struct > do_exit > do_group_exit > get_signal_to_deliver > do_signal > do_notify_resume > int_signal > > Seams to even not work if there's just one (legal) PCI device plugged > in. > > Could somebody please shed some light on this problem? > > > Thank you, > kind regards, > Andreas Hartmann