Re: [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy

[Andreas Hartmann <andihartmann@freenet.de>]

Hello Don!

Am Tue, 13 Dec 2011 01:21:41 +0100
schrieb Andreas Hartmann <andihartmann@01019freenet.de>:

> Am Mon, 12 Dec 2011 13:36:36 -0500
> schrieb Don Dutile <ddutile@redhat.com>:
> 
> > On 12/12/2011 06:15 AM, Andreas Hartmann wrote:
> > > Hello!
> > >
> > > I've got a few questions to a problem, which already was analyzed here
> > > sometime ago:
> > > http://markmail.org/message/dspovwvzp3wtdrf6#query:+page:1+mid:i2oph4xwfmiknt3y+state:results
> > >
> > > My situation is a bit different. I do have two PCI cards (a Linksys wlan
> > > card and an intel e100 card). Each of these cards should be managed in
> > > an own VM. I do have no problems with IRQ sharing (each device does have its own IRQ).
> > >
> > > I'm using linux 3.0.6, kvm 0.15 and libvirt 0.9.7. Mainboard is: GA-990XA-UD3.
> > >
> > >
> > > The problem is: both cards are behind a PCI-PCI bridge:
> > >
> > > 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
> > >
> > >
> > > -[0000:00]-+-00.0  ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
> > >             +-00.2  ATI Technologies Inc Device 5a23
> > >             +-02.0-[01]--+-00.0  ATI Technologies Inc Device 6759
> > >             |            \-00.1  ATI Technologies Inc Device aa90
> > >             +-04.0-[02]----00.0  Device 1b6f:7023
> > >             +-09.0-[03]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
> > >             +-0a.0-[04]----00.0  Device 1b6f:7023
> > >             +-11.0  ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
> > >             +-12.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> > >             +-12.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
> > >             +-13.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> > >             +-13.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
> > >             +-14.0  ATI Technologies Inc SBx00 SMBus Controller
> > >             +-14.1  ATI Technologies Inc SB700/SB800 IDE Controller
> > >             +-14.2  ATI Technologies Inc SBx00 Azalia (Intel HDA)
> > >             +-14.3  ATI Technologies Inc SB700/SB800 LPC host controller
> > >
> > >             +-14.4-[05]--+-06.0  Intel Corporation 82557/8/9/0/1 Ethernet Pro 100
> > >             |            \-07.0  RaLink RT2800 802.11n PCI
> > >
> > You cannot assign two devices behind a (legacy) PCI (not PCIe) bridge to two
> > different VMs because PCI devices dont provide a B:D.F in their transaction headers;
> > In this case, the PCIe-PCI bridge/switch prepends the bridge's B:D.F in front of
> > a transaction that is sourced by 05:06.0 or 05:07.0 .
> > 
> > So, the devices cannot be isolated from each other's (mem) mapping domains in the
> > IOMMU, thus, libvirt prevents this (security) violation from being enabled.
> 
> Ok. If I remove the intel card and put in instead a 32 bit PCIe card
> like TP-Link TG-3468, I could assign each of these two cards to
> different VMs.
> 
> Is this correct?
> 
> [...]
> 
> > > 1. Is there any way to get the devices into different VMs? (I can't put them to another PCI slot as there are just 2 PCI slots on the board.)
> > Only if the two slots are behind different PCIe-PCI bridges .
> > 
> > > 2. Is there any fix to prevent the host crash - maybe in a newer version of kvm or kernel?
> > don't assign the PCI bridge; libvirt does all the proper bind/unbinding of devices
> > for assigned devices, so the above echo steps are unnecessary if you use libvirt.
> 
> If I just remove the intel card, I should be able to assign the wlan
> card to one of my VMs. I'll try that.

Meanwhile I checked this scenario: I removed the intel card and
rebooted. I got the following pci tree:

-[0000:00]-+-00.0  ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
           +-00.2  ATI Technologies Inc Device 5a23
           +-02.0-[01]--+-00.0  ATI Technologies Inc Device 6759
           |            \-00.1  ATI Technologies Inc Device aa90
           +-04.0-[02]----00.0  Device 1b6f:7023
           +-09.0-[03]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
           +-0a.0-[04]----00.0  Device 1b6f:7023
           +-11.0  ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
           +-12.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
           +-12.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
           +-13.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
           +-13.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
           +-14.0  ATI Technologies Inc SBx00 SMBus Controller
           +-14.1  ATI Technologies Inc SB700/SB800 IDE Controller
           +-14.2  ATI Technologies Inc SBx00 Azalia (Intel HDA)
           +-14.3  ATI Technologies Inc SB700/SB800 LPC host controller
           +-14.4-[05]----07.0  RaLink RT2800 802.11n PCI

           +-14.5  ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
           +-15.0-[06]--
           +-16.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
           +-16.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
           +-18.0  Advanced Micro Devices [AMD] Device 1600
           +-18.1  Advanced Micro Devices [AMD] Device 1601
           +-18.2  Advanced Micro Devices [AMD] Device 1602
           +-18.3  Advanced Micro Devices [AMD] Device 1603
           +-18.4  Advanced Micro Devices [AMD] Device 1604
           \-18.5  Advanced Micro Devices [AMD] Device 1605

But this doesn't work, too: I had to manually unbind the bridge (as described in the mail before - it was
not necessary to manually unbind the device itself). On closing the VM,
the complete host crashed with this screen (excerpt):

amd_iommu_domain_destroy
iommu_domain_free
kvm_iommu_unmap_guest
kvm_arch_destroy_vm
kvm_destroy_vm
kvm_vcpu_release
__fput
flip_close
put_files_struct
do_exit
do_group_exit
get_signal_to_deliver
do_signal
do_notify_resume
int_signal

Seams to even not work if there's just one (legal) PCI device plugged
in.

Could somebody please shed some light on this problem?


Thank you,
kind regards,
Andreas Hartmann