Re: [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy

[Andreas Hartmann <andihartmann@freenet.de>]

Hello kvm-list!

I sent the following text to Don with some additional detailed log
files. If somebody else need them too, I can provide them with pn.


Kind regards,
Andreas Hartmann



Andreas Hartmann schrieb:
> Hello Don,
> 
> thank you for your reply!
> 
> I just want to describe in short the two problems I encounter (my
> description was a bit chaotic :-) I hope it's better now :-)). I
> attached details in two files (dmesg.bz2 and lspci.bz2. It's the
> untouched raw output - no unbind or anything else had been done).
> 
> General hardware setup for a quick overview:
> 
> -[0000:00]-+-00.0  ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
>            +-00.2  ATI Technologies Inc Device 5a23
>            +-02.0-[01]--+-00.0  ATI Technologies Inc Device 6759
>            |            \-00.1  ATI Technologies Inc Device aa90
>            +-04.0-[02]----00.0  Device 1b6f:7023
>            +-05.0-[03]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
>            +-09.0-[04]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
>            +-0a.0-[05]----00.0  Device 1b6f:7023
>            +-11.0  ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
>            +-12.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>            +-12.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
>            +-13.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>            +-13.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
>            +-14.0  ATI Technologies Inc SBx00 SMBus Controller
>            +-14.1  ATI Technologies Inc SB700/SB800 IDE Controller
>            +-14.2  ATI Technologies Inc SBx00 Azalia (Intel HDA)
>            +-14.3  ATI Technologies Inc SB700/SB800 LPC host controller
>            +-14.4-[06]----07.0  RaLink RT2800 802.11n PCI
>            +-14.5  ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
>            +-15.0-[07]--
>            +-16.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>            +-16.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
>            +-18.0  Advanced Micro Devices [AMD] Device 1600
>            +-18.1  Advanced Micro Devices [AMD] Device 1601
>            +-18.2  Advanced Micro Devices [AMD] Device 1602
>            +-18.3  Advanced Micro Devices [AMD] Device 1603
>            +-18.4  Advanced Micro Devices [AMD] Device 1604
>            \-18.5  Advanced Micro Devices [AMD] Device 1605
> 
> The relevant devices are (for problem 1):
> 
> 06:07.0 Network controller: RaLink RT2800 802.11n PCI
>         Subsystem: Linksys Device 0067
>         Flags: bus master, slow devsel, latency 32, IRQ 21
>         Memory at fd8e0000 (32-bit, non-prefetchable) [size=64K]
>         Capabilities: [40] Power Management version 3
>         Kernel driver in use: rt2800pci
> 
> with the PCI-PCI bridge above:
> 
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
>         Flags: bus master, VGA palette snoop, 66MHz, medium devsel, latency 64
>         Bus: primary=00, secondary=06, subordinate=06, sec-latency=64
>         I/O behind bridge: 00009000-00009fff
>         Memory behind bridge: fd800000-fd8fffff
>         Prefetchable memory behind bridge: fd700000-fd7fffff
> 
> 
> 
> and (for problem 2):
> 
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
>         Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
>         Flags: bus master, fast devsel, latency 0, IRQ 10
>         I/O ports at de00 [size=256]
>         Memory at fdbff000 (64-bit, non-prefetchable) [size=4K]
>         [virtual] Expansion ROM at fda00000 [disabled] [size=128K]
>         Capabilities: [40] Power Management version 2
>         Capabilities: [48] Vital Product Data
>         Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
>         Capabilities: [60] Express Endpoint, MSI 00
>         Capabilities: [84] Vendor Specific Information: Len=4c <?>
>         Capabilities: [100] Advanced Error Reporting
>         Capabilities: [12c] Virtual Channel
>         Capabilities: [148] Device Serial Number f1-11-00-00-68-4c-e0-00
>         Capabilities: [154] Power Budgeting <?>
> 
> 04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
>         Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
>         Flags: bus master, fast devsel, latency 0, IRQ 10
>         I/O ports at ce00 [size=256]
>         Memory at fd6ff000 (64-bit, prefetchable) [size=4K]
>         Memory at fd6f8000 (64-bit, prefetchable) [size=16K]
>         Capabilities: [40] Power Management version 3
>         Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
>         Capabilities: [70] Express Endpoint, MSI 01
>         Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
>         Capabilities: [d0] Vital Product Data
>         Capabilities: [100] Advanced Error Reporting
>         Capabilities: [140] Virtual Channel
>         Capabilities: [160] Device Serial Number 12-34-56-78-12-34-56-78
> 
> 
> ============================================================================================================================
> 
> 
> 1. problem:
> 
> Wlan RaLink RT2800 802.11n is a legacy PCI device behind
> a PCI-PCI bridge and should be assigned to VM_a, but passthrough doesn't
> work because libvirt claims, that the device or resource would be busy
> (but the device is unbound correctly).
> 
> There is no other device in the tree under the bridge (for details
> please see the attached log file).
> 
> This is the entry in the xml-file I use:
> 
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x06' slot='0x07' function='0x0'/>
>       </source>
>     </hostdev>
> 
> The VM is started with "virsh start VM_a".
> 
> I was able to start VM_a if the PCI-PCI bridge was unbound manually
> before VM_a was started (yes, I really know, that libvirt does the
> unbind job for me - but it doesn't work as expected :-). 
> The bridge itself wasn't assigned to VM_a):
> 
> echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:00:14.4" > /sys/bus/pci/devices/0000:00:14.4/driver/unbind
> 
> Unfortunately, the host is locked up on shutdown of VM_a.
> 
> That's the trace of the locked host:
> 
> amd_iommu_domain_destroy
> iommu_domain_free
> kvm_iommu_unmap_guest
> kvm_arch_destroy_vm
> kvm_destroy_vm
> kvm_vcpu_release
> __fput
> flip_close
> put_files_struct
> do_exit
> do_group_exit
> get_signal_to_deliver
> do_signal
> do_notify_resume
> int_signal
> 
> 
> But I think this should be a valid setup according [1]:
> 
> "Notes:
> VT-d spec specifies that all conventional PCI devices behind a PCIe-to
> PCI/PCI-X bridge or conventional PCI bridge can only be collectively
> assigned to the same guest. PCIe devices do not have this restriction."
> 
> [1] http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
> 
> 
> 
> 
> 2. problem:
> 
> Next I tried to assign the PCIe ethernet card, TP-Link TG-3468
> (device 03:00.0), to VM_b.
> 
> This card does have exactly the same PCIID as the onboard network
> interface. If I put it (or even both ethernet network devices) to VM_b,
> the machine is locked up completely directly on start up of the "virsh
> start VM_b" command.
> 
> Problem seems to be. that the kernel (-> for pci passthrough) can't
> distinguish those two devices because of their same PCIID (there were
> always both devices unbound, even if there should be just one device
> unound).
> 
> The xml-entry is:
> 
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
>       </source>
>     </hostdev>
> 
> If I remove the additional PCIe TP-Link card, I'm able to passthrough
> the onboard ethernet card to VM_b without any problem.
> 
> 
> 
> 
> I hope I could clarify now the two problems I encounter.
> 
> 
> Thank you very much for your time!
> Andreas