Re: [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy

[Don Dutile <ddutile@redhat.com>]

On 12/12/2011 06:15 AM, Andreas Hartmann wrote:
> Hello!
>
> I've got a few questions to a problem, which already was analyzed here
> sometime ago:
> http://markmail.org/message/dspovwvzp3wtdrf6#query:+page:1+mid:i2oph4xwfmiknt3y+state:results
>
> My situation is a bit different. I do have two PCI cards (a Linksys wlan
> card and an intel e100 card). Each of these cards should be managed in
> an own VM. I do have no problems with IRQ sharing (each device does have its own IRQ).
>
> I'm using linux 3.0.6, kvm 0.15 and libvirt 0.9.7. Mainboard is: GA-990XA-UD3.
>
>
> The problem is: both cards are behind a PCI-PCI bridge:
>
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
>
>
> -[0000:00]-+-00.0  ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
>             +-00.2  ATI Technologies Inc Device 5a23
>             +-02.0-[01]--+-00.0  ATI Technologies Inc Device 6759
>             |            \-00.1  ATI Technologies Inc Device aa90
>             +-04.0-[02]----00.0  Device 1b6f:7023
>             +-09.0-[03]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
>             +-0a.0-[04]----00.0  Device 1b6f:7023
>             +-11.0  ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
>             +-12.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>             +-12.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
>             +-13.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>             +-13.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
>             +-14.0  ATI Technologies Inc SBx00 SMBus Controller
>             +-14.1  ATI Technologies Inc SB700/SB800 IDE Controller
>             +-14.2  ATI Technologies Inc SBx00 Azalia (Intel HDA)
>             +-14.3  ATI Technologies Inc SB700/SB800 LPC host controller
>
>             +-14.4-[05]--+-06.0  Intel Corporation 82557/8/9/0/1 Ethernet Pro 100
>             |            \-07.0  RaLink RT2800 802.11n PCI
>
You cannot assign two devices behind a (legacy) PCI (not PCIe) bridge to two
different VMs because PCI devices dont provide a B:D.F in their transaction headers;
In this case, the PCIe-PCI bridge/switch prepends the bridge's B:D.F in front of
a transaction that is sourced by 05:06.0 or 05:07.0 .

So, the devices cannot be isolated from each other's (mem) mapping domains in the
IOMMU, thus, libvirt prevents this (security) violation from being enabled.

>             +-14.5  ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
>             +-15.0-[06]--
>             +-16.0  ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>             +-16.2  ATI Technologies Inc SB700/SB800 USB EHCI Controller
>             +-18.0  Advanced Micro Devices [AMD] Device 1600
>             +-18.1  Advanced Micro Devices [AMD] Device 1601
>             +-18.2  Advanced Micro Devices [AMD] Device 1602
>             +-18.3  Advanced Micro Devices [AMD] Device 1603
>             +-18.4  Advanced Micro Devices [AMD] Device 1604
>             \-18.5  Advanced Micro Devices [AMD] Device 1605
>
> That's what can be seen during boot:
>
> [    0.621901] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
> [    0.621906] AMD-Vi:        mmio-addr: 00000000fec30000
> [    0.622091] AMD-Vi:   DEV_SELECT_RANGE_START  devid: 00:00.0 flags: 00
> [    0.622095] AMD-Vi:   DEV_RANGE_END           devid: 00:00.2
> [    0.622097] AMD-Vi:   DEV_SELECT                      devid: 00:02.0 flags: 00
> [    0.622100] AMD-Vi:   DEV_SELECT_RANGE_START  devid: 01:00.0 flags: 00
> [    0.622102] AMD-Vi:   DEV_RANGE_END           devid: 01:00.1
> [    0.622105] AMD-Vi:   DEV_SELECT                      devid: 00:04.0 flags: 00
> [    0.622107] AMD-Vi:   DEV_SELECT                      devid: 02:00.0 flags: 00
> [    0.622109] AMD-Vi:   DEV_SELECT                      devid: 00:09.0 flags: 00
> [    0.622112] AMD-Vi:   DEV_SELECT                      devid: 03:00.0 flags: 00
> [    0.622114] AMD-Vi:   DEV_SELECT                      devid: 00:0a.0 flags: 00
> [    0.622117] AMD-Vi:   DEV_SELECT                      devid: 04:00.0 flags: 00
> [    0.622119] AMD-Vi:   DEV_SELECT                      devid: 00:11.0 flags: 00
> [    0.622122] AMD-Vi:   DEV_SELECT_RANGE_START  devid: 00:12.0 flags: 00
> [    0.622124] AMD-Vi:   DEV_RANGE_END           devid: 00:12.2
> [    0.622127] AMD-Vi:   DEV_SELECT_RANGE_START  devid: 00:13.0 flags: 00
> [    0.622129] AMD-Vi:   DEV_RANGE_END           devid: 00:13.2
> [    0.622132] AMD-Vi:   DEV_SELECT                      devid: 00:14.0 flags: d7
> [    0.622134] AMD-Vi:   DEV_SELECT                      devid: 00:14.1 flags: 00
> [    0.622137] AMD-Vi:   DEV_SELECT                      devid: 00:14.2 flags: 00
> [    0.622139] AMD-Vi:   DEV_SELECT                      devid: 00:14.3 flags: 00
> [    0.622141] AMD-Vi:   DEV_SELECT                      devid: 00:14.4 flags: 00
> [    0.622144] AMD-Vi:   DEV_ALIAS_RANGE                 devid: 05:00.0 flags: 00 devid_to: 00:14.4
> [    0.622147] AMD-Vi:   DEV_RANGE_END           devid: 05:1f.7
> [    0.622154] AMD-Vi:   DEV_SELECT                      devid: 00:14.5 flags: 00
> [    0.622157] AMD-Vi:   DEV_SELECT                      devid: 00:15.0 flags: 00
> [    0.622159] AMD-Vi:   DEV_SELECT_RANGE_START  devid: 06:00.0 flags: 00
> [    0.622162] AMD-Vi:   DEV_RANGE_END           devid: 06:1f.7
> [    0.622166] AMD-Vi:   DEV_SELECT_RANGE_START  devid: 00:16.0 flags: 00
> [    0.622169] AMD-Vi:   DEV_RANGE_END           devid: 00:16.2
> [    0.622176] pci 0000:00:00.2: can't derive routing for PCI INT A
> [    0.622179] pci 0000:00:00.2: PCI INT A: no GSI
> [    0.622379] pci 0000:00:00.2: irq 40 for MSI/MSI-X
> [    0.622453] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
> [    0.681866] AMD-Vi: Lazy IO/TLB flushing enabled
> [    0.681870] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> [    0.681873] Placing 64MB software IO TLB between ffff8800cbd9d000 - ffff8800cfd9d000
> [    0.681877] software IO TLB at phys 0xcbd9d000 - 0xcfd9d000
>
> Unbind is done like this:
> # PCI bridge
> echo "1002 4384">  /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:00:14.4">  /sys/bus/pci/drivers/pci-stub/unbind
>
> # e100
> echo "8086 1229">  /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:05:06.0">  /sys/bus/pci/drivers/pci-stub/unbind
> echo "0000:05:06.0">  /sys/bus/pci/drivers/pci-stub/bind
> echo "8086 1229">  /sys/bus/pci/drivers/pci-stub/remove_id
>
> # Linksys
> echo "1814 0601">  /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:05:07.0">  /sys/bus/pci/drivers/pci-stub/unbind
> echo "0000:05:07.0">  /sys/bus/pci/drivers/pci-stub/bind
> echo "1814 0601">  /sys/bus/pci/drivers/pci-stub/remove_id
>
> I can start a VM (and can see the devices in the VM) with both devices or just one PCI device (without the bridge). If the machine in this VM is halted, the host crashes as already described here:
> http://markmail.org/message/dspovwvzp3wtdrf6
>
> (The bridge can't be put to the VM because libvirt (I'm using libvirt) doesn't like it.)
>
>
> Unfortunately, it wasn't possible to put the PCI devices to different VMs at the same time (Failed to assign device "hostdev0" : Device or resource busy) - but that's exactly what I need :-(.
>
>
> Now my questions:
>
> 1. Is there any way to get the devices into different VMs? (I can't put them to another PCI slot as there are just 2 PCI slots on the board.)
Only if the two slots are behind different PCIe-PCI bridges .

> 2. Is there any fix to prevent the host crash - maybe in a newer version of kvm or kernel?
don't assign the PCI bridge; libvirt does all the proper bind/unbinding of devices
for assigned devices, so the above echo steps are unnecessary if you use libvirt.

>
>
> Thank you very much for your help,
> kind regards,
> Andreas Hartmann
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html