Re: [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy

[Don Dutile <ddutile@redhat.com>]

On 12/13/2011 03:40 AM, Andreas Hartmann wrote:
> Hello Don!
>
Andreas,

sorry for the delay, I don't follow this email list frequently (read: !daily).

> Am Tue, 13 Dec 2011 01:21:41 +0100
> schrieb Andreas Hartmann<andihartmann@01019freenet.de>:
>
>> Am Mon, 12 Dec 2011 13:36:36 -0500
>> schrieb Don Dutile<ddutile@redhat.com>:
>>
>>> On 12/12/2011 06:15 AM, Andreas Hartmann wrote:
>>>> Hello!
>>>>
>>>> I've got a few questions to a problem, which already was analyzed here
>>>> sometime ago:
>>>> http://markmail.org/message/dspovwvzp3wtdrf6#query:+page:1+mid:i2oph4xwfmiknt3y+state:results
>>>>
>>>> My situation is a bit different. I do have two PCI cards (a Linksys wlan
>>>> card and an intel e100 card). Each of these cards should be managed in
>>>> an own VM. I do have no problems with IRQ sharing (each device does have its own IRQ).
>>>>
>>>> I'm using linux 3.0.6, kvm 0.15 and libvirt 0.9.7. Mainboard is: GA-990XA-UD3.
>>>>
>>>>
>>>> The problem is: both cards are behind a PCI-PCI bridge:
>>>>
>>>> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
>>>>
>>>>
>>>> -[0000:00]-+-00.0  ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
>>>>              +-00.2  ATI Technologies Inc Device 5a23
>>>>              +-02.0-[01]--+-00.0  ATI Technologies Inc Device 6759
>>>>              |            \-00.1  ATI Technologies Inc Device aa90
>>>>              +-04.0-[02]----00.0  Device 1b6f:7023
>>>>              +-09.0-[03]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
>>>>              +-0a.0-[04]----00.0  Device 1b6f:7023
>>>>              +-11.0  ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
>>>>              +-12.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>>>>              +-12.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
>>>>              +-13.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>>>>              +-13.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
>>>>              +-14.0  ATI Technologies Inc SBx00 SMBus Controller
>>>>              +-14.1  ATI Technologies Inc SB700/SB800 IDE Controller
>>>>              +-14.2  ATI Technologies Inc SBx00 Azalia (Intel HDA)
>>>>              +-14.3  ATI Technologies Inc SB700/SB800 LPC host controller
>>>>
>>>>              +-14.4-[05]--+-06.0  Intel Corporation 82557/8/9/0/1 Ethernet Pro 100
>>>>              |            \-07.0  RaLink RT2800 802.11n PCI
>>>>
>>> You cannot assign two devices behind a (legacy) PCI (not PCIe) bridge to two
>>> different VMs because PCI devices dont provide a B:D.F in their transaction headers;
>>> In this case, the PCIe-PCI bridge/switch prepends the bridge's B:D.F in front of
>>> a transaction that is sourced by 05:06.0 or 05:07.0 .
>>>
>>> So, the devices cannot be isolated from each other's (mem) mapping domains in the
>>> IOMMU, thus, libvirt prevents this (security) violation from being enabled.
>>
>> Ok. If I remove the intel card and put in instead a 32 bit PCIe card
>> like TP-Link TG-3468, I could assign each of these two cards to
>> different VMs.
>>
KISS: Don't try to assign legacy PCI devices; just PCIe. ;-)

Where is "TP-Link TG-3468" in lspci output below?  the AMD devices in 18.0->18.5 ???

>> Is this correct?
>>
>> [...]
>>
>>>> 1. Is there any way to get the devices into different VMs? (I can't put them to another PCI slot as there are just 2 PCI slots on the board.)
>>> Only if the two slots are behind different PCIe-PCI bridges .
>>>
>>>> 2. Is there any fix to prevent the host crash - maybe in a newer version of kvm or kernel?
>>> don't assign the PCI bridge; libvirt does all the proper bind/unbinding of devices
>>> for assigned devices, so the above echo steps are unnecessary if you use libvirt.
>>
>> If I just remove the intel card, I should be able to assign the wlan
>> card to one of my VMs. I'll try that.
>
> Meanwhile I checked this scenario: I removed the intel card and
> rebooted. I got the following pci tree:
>
> -[0000:00]-+-00.0  ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
>             +-00.2  ATI Technologies Inc Device 5a23
>             +-02.0-[01]--+-00.0  ATI Technologies Inc Device 6759
>             |            \-00.1  ATI Technologies Inc Device aa90
>             +-04.0-[02]----00.0  Device 1b6f:7023
>             +-09.0-[03]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
>             +-0a.0-[04]----00.0  Device 1b6f:7023
>             +-11.0  ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
>             +-12.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>             +-12.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
>             +-13.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>             +-13.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
>             +-14.0  ATI Technologies Inc SBx00 SMBus Controller
>             +-14.1  ATI Technologies Inc SB700/SB800 IDE Controller
>             +-14.2  ATI Technologies Inc SBx00 Azalia (Intel HDA)
>             +-14.3  ATI Technologies Inc SB700/SB800 LPC host controller
>             +-14.4-[05]----07.0  RaLink RT2800 802.11n PCI
>
>             +-14.5  ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
>             +-15.0-[06]--
>             +-16.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>             +-16.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
>             +-18.0  Advanced Micro Devices [AMD] Device 1600
>             +-18.1  Advanced Micro Devices [AMD] Device 1601
>             +-18.2  Advanced Micro Devices [AMD] Device 1602
>             +-18.3  Advanced Micro Devices [AMD] Device 1603
>             +-18.4  Advanced Micro Devices [AMD] Device 1604
>             \-18.5  Advanced Micro Devices [AMD] Device 1605
>
> But this doesn't work, too: I had to manually unbind the bridge (as described in the mail before - it was
> not necessary to manually unbind the device itself). On closing the VM,
> the complete host crashed with this screen (excerpt):
>

You cannot assign a PCI bridge. In fact, you don't need to for device-assignment.
You just need to assign the device itself; nothing else in the host PCI hierarchy.
.... assuming the device is a PCIe device, and not a graphics card/device.

again, libvirt does all the bind/unbinding you need to do.

You want to do a 'virsh device-attach <dom-id> pci.xml
with pci.xml looking something like:
<hostdev mode='subsystem' type='pci' managed='no'>
   <source>
     <address bus='0' slot='XX' function='0'/>
   </source>
</hostdev>

where XX is replaced by the value of the slot your PCIe device is plugged into.


> amd_iommu_domain_destroy
> iommu_domain_free
> kvm_iommu_unmap_guest
> kvm_arch_destroy_vm
> kvm_destroy_vm
> kvm_vcpu_release
> __fput
> flip_close
> put_files_struct
> do_exit
> do_group_exit
> get_signal_to_deliver
> do_signal
> do_notify_resume
> int_signal
>
> Seams to even not work if there's just one (legal) PCI device plugged
> in.
>
> Could somebody please shed some light on this problem?
>
>
> Thank you,
> kind regards,
> Andreas Hartmann